Devices, Methods, and Graphical User Interfaces for Interaction with an Intensity-Sensitive Input Region

ABSTRACT

An electronic device includes a display and an input region separate from the display. The device detects a first portion of a first input on the input region. In response to detecting the first portion of the first input: if the first portion of the first input meets first criteria that do not require a characteristic intensity of the first input to increase above a first intensity threshold in order to be met, the device displays a precursor object of a user interface object that includes one or more selectable options. While displaying the precursor object, a second portion of the first input is detected on the input region, wherein an intensity of the first input increases above the first intensity threshold. In response to detecting the increase in intensity, the device reveals the user interface object by an amount that varies with a current intensity of the first input.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 62/668,187, filed May 7, 2018, which is incorporated by referenceherein in its entirety.

TECHNICAL FIELD

This relates generally to electronic devices with displays andintensity-sensitive input regions, including but not limited toelectronic devices with displays and intensity-sensitive off-displayinput regions.

BACKGROUND

Many electronic devices with displays include mechanical buttons, suchas mechanical home buttons, volume buttons, and power buttons. Butmechanical buttons provide little, if any, feedback to a user beyond afixed down click and a fixed up click. Some devices have solid statehome buttons on the display-side of the devices to replace themechanical home buttons. A display-side solid state home button iscoupled with tactile output generators that generate tactile outputs tosimulate various types of mechanical button clicks when the solid statehome button is activated by a press input. The solid state home buttonis a close analog of a mechanical home button and is designated for avery limited set of functions triggered by a few types of press inputs.Display-side solid state buttons also take up valuable space on thedisplay-side of the device, reducing available display area for visualinformation on a portable electronic device.

SUMMARY

Accordingly, there is a need for electronic devices with improvedmethods and interfaces for providing visual, haptic, and/or audiofeedback during interaction with an off-display input region (e.g., anintensity-sensitive side button or surface), which make manipulation ofuser interfaces more efficient and intuitive for a user. There is also aneed for electronic devices with improved methods and interfaces forproviding additional functionality without cluttering up the displaywith more virtual controls or crowding the surface of the device withmore hardware controls. Such methods and interfaces optionallycomplement or replace conventional methods for providing additionalfunction controls and providing feedback during interaction with adevice using a button. Such methods and interfaces reduce the number andextent of the inputs required from a user by helping the user tounderstand the connection between provided inputs and device responsesto the inputs and reducing user input mistakes, thereby creating a moreefficient human-machine interface.

The above deficiencies and other problems associated with userinterfaces for electronic devices with buttons are reduced or eliminatedby the disclosed devices, which include one or more intensity-sensitiveinput regions (e.g., off-display intensity-sensitive buttons or surfaceslocated on peripheral sides of the device). In some embodiments, thedevice is a desktop computer. In some embodiments, the device isportable (e.g., a notebook computer, tablet computer, or handhelddevice). In some embodiments, the device is a personal electronic device(e.g., a wearable electronic device, such as a watch). In someembodiments, the device has a touchpad. In some embodiments, the devicehas a touch-sensitive display (also known as a “touch screen” or“touch-screen display”). In some embodiments, the device has a graphicaluser interface (GUI), one or more processors, memory and one or moremodules, programs or sets of instructions stored in the memory forperforming multiple functions. In some embodiments, the user interactswith the GUI primarily through stylus and/or finger contacts andgestures on the touch-sensitive surface. In some embodiments, thefunctions optionally include image editing, drawing, presenting, wordprocessing, spreadsheet making, game playing, telephoning, videoconferencing, e-mailing, instant messaging, workout support, digitalphotographing, digital videoing, web browsing, digital music playing,note taking, and/or digital video playing. Executable instructions forperforming these functions are, optionally, included in a non-transitorycomputer readable storage medium or other computer program productconfigured for execution by one or more processors.

In accordance with some embodiments, a method is performed at anelectronic device with a display and a first input region that isseparate from the display. The method includes: detecting a firstportion of a first input on the first input region; in response todetecting the first portion of the first input: in accordance with adetermination that the first portion of the first input meets firstcriteria, wherein the first criteria include a set of criteria that aremet when the first input is detected on the first input region without acharacteristic intensity of the first input increasing above a firstintensity threshold, providing a first output that indicates a currentstate of a first attribute of the device without altering the firstattribute of the device; detecting a second portion of the input,including detecting an increase in intensity of the first input; and inresponse to detecting the second portion of the first input: inaccordance with a determination that the second portion of the firstinput meets second criteria before an end of the first input isdetected, wherein the second criteria include a set of criteria that aremet when the characteristic intensity of the first input increases abovethe first intensity threshold, performing a first operation that altersthe first attribute of the device; and in accordance with adetermination that the second portion of the first input does not meetthe second criteria, forgoing performing the first operation that altersthe first attribute of the device.

In accordance with some embodiments, a method is performed at anelectronic device with a display, and a first input region that isseparate from the display. The method includes: detecting a firstportion of a first input on the first input region; in response todetecting the first portion of the first input on the first inputregion: in accordance with a determination that the first portion of thefirst input meets first criteria, wherein the first criteria include aset of criteria that are met when the first input is detected on thefirst input region without a characteristic intensity of the first inputincreasing above a first intensity threshold, displaying, on thedisplay, a precursor object of a user interface object that includes oneor more selectable options; while displaying the precursor object on thedisplay, detecting a second portion of the first input on first inputregion, including detecting an increase in intensity of the first inputabove the first intensity threshold; and in response to detecting theincrease in intensity of the first input during the second portion ofthe first input, revealing the user interface object on the display byan amount that varies in accordance with a current intensity of thefirst input.

In accordance with some embodiments, a method is performed at anelectronic device with a display, an input region that is distinct fromthe display and includes a first sub-region and a second sub-region. Themethod includes: detecting an input on the input region; in response todetecting the input on the input region: in accordance with adetermination that the input meets first criteria, wherein the firstcriteria require an increase in intensity of the input above a firstintensity threshold while the input is detected in the first sub-regionof the input region, in order for the first criteria to be met,modifying a first parameter of the device in a first manner; inaccordance with a determination that the input meets second criteriathat are distinct from the first criteria, wherein the second criteriarequire an increase in intensity of the input above the first intensitythreshold while the input is detected in the second sub-region of theinput region that is distinct from the first sub-region, in order forthe second criteria to be met, modifying the first parameter of thedevice in a second manner that is distinct from the first manner; and inaccordance with a determination that the input did not increase abovethe first intensity threshold, forgoing modifying the first parameter ofthe device in the first manner and forgoing modifying the firstparameter of the device in the second manner.

In accordance with some embodiments, a method is performed at anelectronic device with a display, one or more cameras, and a pluralityof tactile output generators including a first tactile output generatorat a first location in the device and a second tactile output generatorat a second location in the device. The method includes: while thedevice is in a media capture mode of operation, performing a respectivetask: detecting that one or more objects in a field of view of the oneor more cameras of the device meet first criteria; and in response todetecting that the one or more objects in the field of view of the oneor more cameras meet the first criteria, generating, with one or more ofthe plurality of tactile output generators, a tactile output promptingmovement of the field of view of the one or more cameras in a respectivedirection, wherein generating the tactile output prompting movement ofthe field of view of the one or more cameras in the respective directionincludes: in accordance with a determination, based on the one or moreobjects in the field of view of the camera, that the field of view ofthe camera needs to be shifted in a first direction to continueperforming the respective task, generating a first tactile output thatprompts movement of the field of view of the one or more cameras in thefirst direction; and in accordance with a determination, based on theone or more objects in the field of view of the camera, that the fieldof view of the camera needs to be shifted in a second direction,different from the first direction, to continue performing therespective task, generating a second tactile output, different from thefirst tactile output, that prompts movement of the field of view of theone or more cameras in the second direction.

In accordance with some embodiments, a method is performed at anelectronic device with a display, and an input region that is associatedwith a tactile output generator. The method includes: detecting an inputon the input region; in response to detecting the input on the inputregion: in accordance with a determination that first criteria are metby the input on the input region, wherein the first criteria include arequirement that is met when a first input metric exceeds a firstthreshold in order for the first criteria to be met, generating a firsttactile output with the tactile output generator associated with theinput region; and after generating the first tactile output: inaccordance with a determination that the input meets second criteria,wherein the second criteria include a requirement that is met when thefirst input metric exceeds a second threshold that is greater than thefirst threshold in order for the second criteria to be met, performingan operation; and in accordance with a determination that a cancelationinput has been detected prior to the input meeting the second criteria,forgoing performance of the operation; and in accordance with adetermination that the first criteria are not met by the input on theinput region, forgoing generating the first tactile output.

In accordance with some embodiments, a method is performed at anelectronic device with a display, an input region that is distinct fromthe display, and one or more tactile output generators for generatingtactile outputs. The method includes: detecting a press input on theinput region with a finger; and in response to detecting the press inputon the input region: in accordance with a determination that the inputregion is not separated from the finger by an interstitial material,generating a first tactile output; and in accordance with adetermination that the input region is separated from the finger by afirst interstitial material, generating a second tactile output that isdifferent from the first tactile output.

In accordance with some embodiments, a method is performed at anelectronic device with a display that is disposed on a front-side of thedevice, an input region that is disposed on one or more peripheral-sidesof the device adjacent to the front-side of the device, and one or moretactile output generators for generating tactile outputs. The methodincludes: detecting an input on the input region; in response todetecting the input on the input region: in accordance with adetermination that the input by the contact on the input region meetsfirst criteria, wherein the first criteria require an increase inintensity of the input above a first intensity threshold in order forthe first criteria to be met, performing a first operation andgenerating a first tactile output with a first tactile output profile;and in accordance with a determination that the input on the inputregion meets second criteria that are distinct from the first criteria,wherein the second criteria require that the input includes movement ofa contact along a first direction across the input region in order forthe second criteria to be met, performing a second operation differentfrom the first operation and generating a second tactile output with asecond tactile output profile that is distinct from the first tactileoutput profile.

In accordance with some embodiments, an electronic device includes adisplay, a touch-sensitive surface, one or more intensity-sensitiveinput regions (e.g., side buttons), one or more sensors to detectintensities of contacts with the touch-sensitive surface and the inputregions, optionally, one or more tactile output generators forgenerating localized tactile outputs at the input regions and/or wholedevice tactile outputs throughout multiple regions of the device, one ormore processors, memory, and one or more programs; the one or moreprograms are stored in the memory and configured to be executed by theone or more processors and the one or more programs include instructionsfor performing or causing performance of the operations of any of themethods described herein. In accordance with some embodiments, acomputer readable storage medium has stored therein instructions whichwhen executed by an electronic device with a display, a touch-sensitivesurface, one or more intensity-sensitive input regions (e.g., sidebuttons), one or more sensors to detect intensities of contacts with thetouch-sensitive surface and the input regions, and optionally, one ormore tactile output generators for generating localized tactile outputsat the input regions and/or whole device tactile outputs throughoutmultiple regions of the device, cause the device to perform or causeperformance of the operations of any of the methods described herein. Inaccordance with some embodiments, a graphical user interface on anelectronic device with a display, a touch-sensitive surface, one or moreintensity-sensitive input regions (e.g., side buttons), one or moresensors to detect intensities of contacts with the touch-sensitivesurface, optionally, one or more tactile output generators forgenerating localized tactile outputs at the input regions and/or wholedevice tactile outputs throughout multiple regions of the device, amemory, and one or more processors to execute one or more programsstored in the memory includes one or more of the elements displayed inany of the methods described herein, which are updated in response toinputs, as described in any of the methods described herein. Inaccordance with some embodiments, an electronic device includes: adisplay, a touch-sensitive surface, one or more intensity-sensitiveinput regions (e.g., side buttons), one or more sensors to detectintensities of contacts with the touch-sensitive surface, optionally,one or more tactile output generators for generating localized tactileoutputs at the input regions and/or whole device tactile outputsthroughout multiple regions of the device; and means for performing orcausing performance of the operations of any of the methods describedherein. In accordance with some embodiments, an information processingapparatus, for use in an electronic device with a display and atouch-sensitive surface, one or more intensity-sensitive input regions(e.g., side buttons), one or more sensors to detect intensities ofcontacts with the touch-sensitive surface, optionally, one or moretactile output generators for generating localized tactile outputs atthe input regions and/or whole device tactile outputs throughoutmultiple regions of the device, includes means for performing or causingperformance of the operations of any of the methods described herein.

Thus, electronic devices with displays, touch-sensitive surfaces, one ormore intensity-sensitive input regions (e.g., side buttons), one or moresensors to detect intensities of contacts with the touch-sensitivesurface, optionally, one or more tactile output generators forgenerating localized tactile outputs at the input regions and/or wholedevice tactile outputs throughout multiple regions of the device,optionally one or more device orientation sensors, and optionally anaudio system, are provided with improved methods and interfaces forproviding feedback to a user during interaction with an off-displayinput region, thereby increasing the effectiveness, efficiency, and usersatisfaction with such devices. Such methods and interfaces maycomplement or replace conventional methods for facilitating interactionswith the device and providing haptic feedback to a user.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1A is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments.

FIG. 1C is a block diagram illustrating a tactile output module inaccordance with some embodiments.

FIG. 2A illustrates a portable multifunction device having a touchscreen in accordance with some embodiments.

FIGS. 2B-2C show exploded views of an intensity-sensitive input devicein accordance with some embodiments.

FIGS. 2D-2F show different views of an intensity-sensitive input devicein accordance with some embodiments.

FIG. 3 is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIG. 4A illustrates a portable multifunction device with one or moreintensity-sensitive off-display input regions on one or more peripheralsides of the device, and a touch-screen display showing an example homescreen user interface with a menu of application launch iconscorresponding to different applications, in accordance with someembodiments.

FIG. 4B illustrates a multifunction device with a touch-sensitivesurface that is separate from the display in accordance with someembodiments.

FIGS. 4C-4E illustrate examples of dynamic intensity thresholds inaccordance with some embodiments.

FIGS. 4F1-4F4 and 4G1-4G4 illustrate a set of sample tactile outputpatterns in accordance with some embodiments.

FIGS. 4H-4J illustrate example haptic audio output patterns versus timethat are used in conjunction with tactile outputs to simulate buttonclicks in accordance with some embodiments.

FIGS. 4K-4Q illustrate example combinations of tactile output patternsand haptic audio output patterns versus time in accordance with someembodiments.

FIGS. 5A1-5A8 illustrate exemplary user interfaces for providing hapticand visual feedback for button interactions corresponding to queryingand/or modifying a current state of a device attribute in accordancewith some embodiments.

FIGS. 5B1-5B11 illustrate exemplary user interfaces for providing hapticand visual feedback for button interactions corresponding to modifying adevice parameter in different manners in accordance with someembodiments.

FIGS. 5C1-5C20 and 5D1-5D24 illustrate exemplary user interfaces forproviding haptic and visual feedback for different types of buttoninteractions corresponding to presenting and interacting with aretractable menu in accordance with some embodiments.

FIGS. 5E1-5E5 and 5F1-5F6 illustrate exemplary user interfaces forproviding haptic and visual feedback for guiding device movement toperform a respective task when the device is in a media capture mode ofoperation in accordance with some embodiments.

FIGS. 5G1-5G5 and 5H1-5H5 illustrate exemplary user interfaces forproviding haptic and visual feedback for warning a user of impendingperformance of an operation in accordance with some embodiments.

FIGS. 5I1-5I6 illustrate providing, for each respective type of buttoninteraction, tactile outputs that have varying tactile output profilesdepending on the presence of an interstitial material between the inputregion and the user's finger in accordance with some embodiments.

FIGS. 6A-6D are flow diagrams illustrating a method of querying andmodifying a device attribute in accordance with some embodiments.

FIGS. 7A-7E are flow diagrams illustrating a method of presenting andinteracting with a retractable menu in accordance with some embodiments.

FIGS. 8A-8D are flow diagrams illustrating a method of modifying adevice parameter in different manners in accordance with someembodiments.

FIGS. 9A-9C are flow diagrams illustrating a method of guiding devicemovement in accordance with some embodiments.

FIGS. 10A-10C are flow diagrams illustrating a method of generatingwarning feedback for an impending operation in accordance with someembodiments.

FIGS. 11A-11B are flow diagrams illustrating a method of adjustingtactile outputs for an input on an input region depending on whetherthere is an interstitial material present between the input region and auser's finger detected on the input region in accordance with someembodiments.

FIGS. 12A-12E are flow diagrams illustrating a method of providingdifferent tactile outputs for distinct types of inputs detected on anoff-display input region disposed on a peripheral side of the device inaccordance with some embodiments.

DESCRIPTION OF EMBODIMENTS

Many electronic devices with displays include mechanical buttons, suchas mechanical home buttons, for triggering device functions andnavigating between user interfaces. Some devices have solid state homebuttons that mimics mechanical home buttons by providing localizedtactile outputs in response to press inputs detected on the solid statehome buttons. But mechanical buttons and solid state home button aresuitable for activating only a couple of fixed functions, and providelittle feedback beyond simple button clicks. Methods described hereinprovide visual, haptic, and/or audio feedback during a variety of userinteractions with an intensity-sensitive off-screen input region (e.g.,a side button or surface) that make manipulation of user interfaces moreefficient and intuitive for a user.

The methods, devices, and GUIs described herein providing novel andimproved ways of triggering device functions and using haptic and visualfeedback to improve device-user interactions, including:

-   -   querying and optionally modifying a device attribute (e.g.,        presence of unread notifications, current volume, etc.) in        accordance with off-display button interactions, and providing        corresponding haptic feedback, in lieu of or in conjunction with        visual feedback;    -   presenting and interacting with a retractable menu in accordance        with various types of touch inputs detected on an off-display        input region (e.g., detected in the absence of and/or in        conjunction with inputs detected on a touch-screen displaying        the menu) and providing corresponding haptic feedback in        conjunction with visual feedback;    -   providing haptic feedback for modifying a device parameter in        different manners depending on which sub-portions of an        off-display input region are activated by the inputs;    -   providing directional haptic feedback at the intensity-sensitive        input regions to guide device movement to continue performance        of a respective task when a device is in a media capture mode of        operation;    -   generating warning feedback for an impending operation through        coordinated operations of multiple feedback channels (e.g.,        localized tactile outputs on the off-display input regions,        whole device tactile outputs, audio outputs, visual outputs,        etc.);    -   providing consistent haptic feedback to a user under a variety        of operating conditions by adjusting a respective tactile output        profile of tactile outputs generated for an input on an input        region depending on whether there is an interstitial material        present between the input region and a user's finger detected on        the input region; and    -   providing tactile outputs with different tactile output profiles        for distinct types of touch inputs detected on an off-display        input region disposed on a peripheral side of the device.

Example Devices

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first contactcould be termed a second contact, and, similarly, a second contact couldbe termed a first contact, without departing from the scope of thevarious described embodiments. The first contact and the second contactare both contacts, but they are not the same contact, unless the contextclearly indicates otherwise.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting,”depending on the context. Similarly, the phrase “if it is determined” or“if [a stated condition or event] is detected” is, optionally, construedto mean “upon determining” or “in response to determining” or “upondetecting [the stated condition or event]” or “in response to detecting[the stated condition or event],” depending on the context.

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as PDA and/ormusic player functions. Example embodiments of portable multifunctiondevices include, without limitation, the iPhone®, iPod Touch®, and iPad®devices from Apple Inc. of Cupertino, Calif. Other portable electronicdevices, such as laptops or tablet computers with touch-sensitivesurfaces (e.g., touch-screen displays and/or touchpads), are,optionally, used. It should also be understood that, in someembodiments, the device is not a portable communications device, but isa desktop computer with a touch-sensitive surface (e.g., a touch-screendisplay and/or a touchpad).

In the discussion that follows, an electronic device that includes adisplay, a touch-sensitive surface, and one or more off-displayintensity-sensitive input regions is described. It should be understood,however, that the electronic device optionally includes one or moreother physical user-interface devices, such as a physical keyboard, amouse and/or a joystick.

The device typically supports a variety of applications, such as one ormore of the following: a note taking application, a drawing application,a presentation application, a word processing application, a websitecreation application, a disk authoring application, a spreadsheetapplication, a gaming application, a telephone application, a videoconferencing application, an e-mail application, an instant messagingapplication, a workout support application, a photo managementapplication, a digital camera application, a digital video cameraapplication, a web browsing application, a digital music playerapplication, and/or a digital video player application.

The various applications that are executed on the device optionally useat least one common physical user-interface device, such as thetouch-sensitive surface or the one or more intensity-sensitiveoff-display input regions. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the deviceare, optionally, adjusted and/or varied from one application to the nextand/or within a respective application. In this way, a common physicalarchitecture (such as the touch-sensitive surface or intensity-sensitiveoff-display input regions) of the device optionally supports the varietyof applications with user interfaces that are intuitive and transparentto the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 1A is a block diagram illustratingportable multifunction device 100 with touch-sensitive display system112 in accordance with some embodiments. Touch-sensitive display system112 is sometimes called a “touch screen” for convenience, and issometimes simply called a touch-sensitive display. Device 100 includesmemory 102 (which optionally includes one or more computer readablestorage mediums), memory controller 122, one or more processing units(CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry110, speaker 111, microphone 113, input/output (I/O) subsystem 106,other input or control devices 116, and external port 124. Device 100optionally includes one or more optical sensors 164. Device 100optionally includes one or more intensity sensors 165 for detectingintensities of contacts on device 100 (e.g., a touch-sensitive surfacesuch as touch-sensitive display system 112 of device 100). Device 100includes one or more tactile output generators 167 for generatingtactile outputs on device 100 (e.g., generating tactile outputs on atouch-sensitive surface such as touch-sensitive display system 112 ofdevice 100 or touchpad 355 of device 300). These components optionallycommunicate over one or more communication buses or signal lines 103.

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display, a trackpad, anintensity-sensitive side button, or a solid state button) is,optionally, interpreted by the user as a “down click” or “up click” of aphysical actuator button, a “detent” of a physical scroll-wheel or dial,and/or a “snap” of a mechanical switch or toggle, etc. In some cases, auser will feel a tactile sensation, such as an “down click” or “upclick”, a “detent”, or a “snap”, even when there is no movement of aninput region that is physically pressed or swiped by the user's fingermovements, before the movement of the input region is detected, and/orwhen the movement of the input region is drastically different from themovement that a physical mechanical button, scroll-wheel or dial, orswitch would undergo to produce the tactile sensations that the userperceives through the input region. As another example, movement of thetouch-sensitive surface (e.g., a touch-sensitive display, a trackpad, anintensity-sensitive side button, or a solid state button) is,optionally, interpreted or sensed by the user as “roughness” of thetouch-sensitive surface, even when there is no change in smoothness ofthe touch-sensitive surface. While such interpretations of touch by auser will be subject to the individualized sensory perceptions of theuser, there are many sensory perceptions of touch that are common to alarge majority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”, “detent”, or “flip and snap of atoggle or switch”), unless otherwise stated, the generated tactileoutput corresponds to physical displacement of the device or a componentthereof that will generate the described sensory perception for atypical (or average) user. Using tactile outputs to provide hapticfeedback to a user enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device) which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, a tactile output pattern specifies characteristicsof a tactile output, such as the amplitude of the tactile output, theshape of a movement waveform of the tactile output, the frequency of thetactile output, the mode of motion of the tactile output (e.g., linearoscillations in x direction, y-direction, or z-direction; or angularoscillations around x-axis, y-axis, or z-axis, etc.), and/or theduration of the tactile output.

When tactile outputs with different tactile output patterns aregenerated by a device, the tactile outputs may invoke different hapticsensations in a user holding or touching the device and/or the inputregion. The device optionally includes one or more whole device tactileoutput generators (e.g., a moveable mass that is coupled to the housingof the device) to generate tactile outputs at many different locationson the device at the same time. The device may also include one or morelocalized tactile output generators (e.g., a surface oscillator thatoscillates or vibrates around a fixed pivot in various directionsunderneath the input region or other tactile output generator that iscapable of generating tactile outputs that are directed specificallytoward the input region) to generate localized tactile outputs. Thelocalized vibrations have varying amplitudes at different locations onthe device with greater amplitudes on the input region or a sub-portionthereof where a user's finger will typically rest while interacting withthe input region and with lower amplitudes at other places (e.g.,outside of the input region) where a user's hand will not typically restwhile interacting with the input region.

While the sensation of the user is based on the user's perception of thetactile output, most users will be able to identify changes in waveform,frequency, mode of motion, and amplitude of tactile outputs generated bythe device. Thus, the waveform, frequency, mode of motion, and amplitudecan be adjusted to indicate to the user that different types of inputshave been detected on the input region (e.g., an intensity-sensitiveside button, a solid state button, a touch pad, or a touch-screen),and/or different operations have been performed. As such, tactileoutputs with tactile output patterns that are designed, selected, and/orengineered to simulate characteristics (e.g., size, material, weight,stiffness, smoothness, etc.); behaviors (e.g., oscillation,displacement, acceleration, rotation, expansion, etc.); and/orinteractions (e.g., collision, adhesion, repulsion, attraction,friction, etc.) of objects in a given environment (e.g., a userinterface that includes graphical features and objects, a simulatedphysical environment with virtual boundaries and virtual objects, a realphysical environment with physical boundaries and physical objects,and/or a combination of any of the above) will, in some circumstances,provide helpful feedback to users that reduces input errors andincreases the efficiency of the user's operation of the device.Additionally, tactile outputs are, optionally, generated to correspondto feedback that is unrelated to a simulated physical characteristic,such as an input threshold or a selection of an object. Such tactileoutputs will, in some circumstances, provide helpful feedback to usersthat reduces input errors and increases the efficiency of the user'soperation of the device.

In some embodiments, a tactile output with a suitable tactile outputpattern serves as a cue for the occurrence of an event of interest in auser interface or behind the scenes in a device. Examples of the eventsof interest include activation of an affordance (e.g., a real or virtualbutton, or toggle switch) provided on the device or in a user interface,success or failure of a requested operation, reaching or crossing aboundary in a user interface, entry into a new state, switching of inputfocus between objects, activation of a new mode, reaching or crossing aninput threshold, detection or recognition of a type of input or gesture,etc. In some embodiments, tactile outputs are provided to serve as awarning or an alert for an impending event or outcome that would occurunless a redirection or interruption input is timely detected. Tactileoutputs are also used in other contexts to enrich the user experience,improve the accessibility of the device to users with visual or motordifficulties or other accessibility needs, and/or improve efficiency andfunctionality of the user interface and/or the device.

Tactile outputs are optionally accompanied with audio outputs and/orvisible user interface changes, which further enhance a user'sexperience when the user interacts with a user interface and/or thedevice, and facilitate better conveyance of information regarding thestate of the user interface and/or the device, and which reduce inputerrors and increase the efficiency of the user's operation of thedevice. In some embodiments, an accompanying audio enhances the effectof the tactile output and makes the haptic sensation experienced by auser more salient and/or realistic. The tactile output profile of atactile output is optionally extended to include an audio profile for anaccompanying audio output that is generated to supplement the tactileoutput. The audio profile for an accompanying audio output of a tactileoutput includes characteristics of the audio output such as timing(e.g., timing offsets from the corresponding tactile output), amplitude(e.g., amplitude specified in predefined values or a ratio orcorrelation between the amplitude of the audio output and one or morecharacteristics of the tactile output), frequency (e.g., frequencyspecified in predefined values or in terms of a relationship with one ormore characteristics of the tactile output), shape of a waveform (e.g.,a predefined waveform or a waveform specified in terms of a relationshipwith one or more characteristics of the tactile output). The accompanyaudio output for a tactile output is distinct from regular device audiooutput (e.g., audio alerts or media output generated by the deviceindependent of generation of a tactile output).

In some embodiments, multiple different tactile output generators (e.g.,whole device tactile output generators, and/or localized tactile outputgenerators located at different locations on the device) coordinatetheir individual outputs (e.g., with timing coordination, amplitudecoordination, and wave pattern coordination, or a combination of theabove) to convey a sense of direction around the device (e.g., up, down,left, right, front, back, clockwise, counter-clockwise, etc.), whichfurther enhance a user's experience when the user interacts with a userinterface and/or the device, and facilitate better conveyance ofinformation regarding the state of the user interface and/or the device,and which reduce input errors and increase the efficiency of the user'soperation of the device. In some embodiments, the tactile output profileof a tactile output to be generated by a respective tactile outputgenerator is extended to identify other tactile output generators andspecify the respective manner by which the respective tactile outputgenerator will coordinate with these other tactile output generators togenerate coordinated tactile outputs for a respective purpose (e.g.,indicate a direction to a user).

FIG. 4F1 provides a set of sample tactile output patterns that may beused, either individually or in combination, either as is or through oneor more transformations (e.g., modulation, amplification, truncation,etc.), to create suitable haptic feedback in various scenarios and forvarious purposes, such as those mentioned above and those described withrespect to the user interfaces and methods discussed herein. FIGS.4F2-4F4 are expanded views of the graphs shown in FIG. 4F1. This exampleof a palette of tactile outputs shows how a set of three waveforms andeight frequencies can be used to produce an array of tactile outputpatterns. In addition to the tactile output patterns shown in thisfigure, each of these tactile output patterns is optionally adjusted inamplitude by changing a gain value for the tactile output pattern, asshown, for example for FullTap 80 Hz, FullTap 200 Hz, MiniTap 80 Hz,MiniTap 200 Hz, MicroTap 80 Hz, and MicroTap 200 Hz in FIG. 4G1, whichare each shown with variants having a gain of 1.0, 0.75, 0.5, and 0.25.FIGS. 4G2-4G4 are expanded views of the graphs shown in FIG. 4G1. Asshown in FIG. 4G1, changing the gain of a tactile output pattern changesthe amplitude of the pattern without changing the frequency of thepattern or changing the shape of the waveform. In some embodiments,changing the frequency of a tactile output pattern also results in alower amplitude as some tactile output generators are limited by howmuch force can be applied to the moveable mass and thus higher frequencymovements of the mass are constrained to lower amplitudes to ensure thatthe acceleration needed to create the waveform does not require forceoutside of an operational force range of the tactile output generator(e.g., the peak amplitudes of the FullTap at 230 Hz, 270 Hz, and 300 Hzare lower than the amplitudes of the FullTap at 80 Hz, 100 Hz, 125 Hz,and 200 Hz).

In FIG. 4F1, each column shows tactile output patterns that have aparticular waveform. The waveform of a tactile output pattern representsthe pattern of physical displacements relative to a neutral position(e.g., x_(zero)) versus time that a moveable mass goes through togenerate a tactile output with that tactile output pattern. For example,a first set of tactile output patterns shown in the left column in FIG.4F1 (e.g., tactile output patterns of a “FullTap”) each have a waveformthat includes an oscillation with two complete cycles (e.g., anoscillation that starts and ends in a neutral position and crosses theneutral position three times). A second set of tactile output patternsshown in the middle column in FIG. 4F1 (e.g., tactile output patterns ofa “MiniTap”) each have a waveform that includes an oscillation thatincludes one complete cycle (e.g., an oscillation that starts and endsin a neutral position and crosses the neutral position one time). Athird set of tactile output patterns shown in the right column in FIG.4F1 (e.g., tactile output patterns of a “MicroTap”) each have a waveformthat includes an oscillation that include one-half of a complete cycle(e.g., an oscillation that starts and ends in a neutral position anddoes not cross the neutral position). The waveform of a tactile outputpattern also includes a start buffer and an end buffer that representthe gradual speeding up and slowing down of the moveable mass at thestart and at the end of the tactile output. The example waveforms shownin FIG. 4F1-4G4 include x_(min) and x_(max) values which represent themaximum and minimum extent of movement of the moveable mass. For largerelectronic devices with larger moveable masses, there may be larger orsmaller minimum and maximum extents of movement of the mass. The exampleshown in FIGS. 4F1-4G4 describes movement of a mass in 1 dimension,however similar principles would also apply to movement of a moveablemass in two or three dimensions.

As shown in FIG. 4F1, each tactile output pattern also has acorresponding characteristic frequency that affects the “pitch” of ahaptic sensation that is felt by a user from a tactile output with thatcharacteristic frequency. For a continuous tactile output, thecharacteristic frequency represents the number of cycles that arecompleted within a given period of time (e.g., cycles per second) by themoveable mass of the tactile output generator. For a discrete tactileoutput, a discrete output signal (e.g., with 0.5, 1, or 2 cycles) isgenerated, and the characteristic frequency value specifies how fast themoveable mass needs to move to generate a tactile output with thatcharacteristic frequency. As shown in FIG. 4F1, for each type of tactileoutput (e.g., as defined by a respective waveform, such as FullTap,MiniTap, or MicroTap), a higher frequency value corresponds to fastermovement(s) by the moveable mass, and hence, in general, a shorter timeto complete the tactile output (e.g., including the time to complete therequired number of cycle(s) for the discrete tactile output, plus astart and an end buffer time). For example, a FullTap with acharacteristic frequency of 80 Hz takes longer to complete than FullTapwith a characteristic frequency of 100 Hz (e.g., 35.4 ms vs. 28.3 ms inFIG. 4F1). In addition, for a given frequency, a tactile output withmore cycles in its waveform at a respective frequency takes longer tocomplete than a tactile output with fewer cycles its waveform at thesame respective frequency. For example, a FullTap at 150 Hz takes longerto complete than a MiniTap at 150 Hz (e.g., 19.4 ms vs. 12.8 ms), and aMiniTap at 150 Hz takes longer to complete than a MicroTap at 150 Hz(e.g., 12.8 ms vs. 9.4 ms). However, for tactile output patterns withdifferent frequencies this rule may not apply (e.g., tactile outputswith more cycles but a higher frequency may take a shorter amount oftime to complete than tactile outputs with fewer cycles but a lowerfrequency, and vice versa). For example, at 300 Hz, a FullTap takes aslong as a MiniTap (e.g., 9.9 ms).

As shown in FIG. 4F1, a tactile output pattern also has a characteristicamplitude that affects the amount of energy that is contained in atactile signal, or a “strength” of a haptic sensation that may be feltby a user through a tactile output with that characteristic amplitude.In some embodiments, the characteristic amplitude of a tactile outputpattern refers to an absolute or normalized value that represents themaximum displacement of the moveable mass from a neutral position whengenerating the tactile output. In some embodiments, the characteristicamplitude of a tactile output pattern is adjustable, e.g., by a fixed ordynamically determined gain factor (e.g., a value between 0 and 1), inaccordance with various conditions (e.g., customized based on userinterface contexts and behaviors) and/or preconfigured metrics (e.g.,input-based metrics, and/or user-interface-based metrics). In someembodiments, an input-based metric (e.g., an intensity-change metric oran input-speed metric) measures a characteristic of an input (e.g., arate of change of a characteristic intensity of a contact in a pressinput or a rate of movement of the contact across a touch-sensitivesurface) during the input that triggers generation of a tactile output.In some embodiments, a user-interface-based metric (e.g., aspeed-across-boundary metric) measures a characteristic of a userinterface element (e.g., a speed of movement of the element across ahidden or visible boundary in a user interface) during the userinterface change that triggers generation of the tactile output. In someembodiments, the characteristic amplitude of a tactile output patternmay be modulated by an “envelope” and the peaks of adjacent cycles mayhave different amplitudes, where one of the waveforms shown above isfurther modified by multiplication by an envelope parameter that changesover time (e.g., from 0 to 1) to gradually adjust amplitude of portionsof the tactile output over time as the tactile output is beinggenerated.

Although specific frequencies, amplitudes, and waveforms are representedin the sample tactile output patterns in FIG. 4F1 for illustrativepurposes, tactile output patterns with other frequencies, amplitudes,and waveforms may be used for similar purposes. For example, waveformsthat have between 0.5 to 4 cycles can be used. Other frequencies in therange of 60 Hz-400 Hz may be used as well.

FIGS. 4H-4J illustrate example haptic audio output patterns versus timethat are used in conjunction with tactile outputs to simulate buttonclicks in accordance with some embodiments.

FIGS. 4K-4Q illustrate example combinations of tactile output patternsand haptic audio output patterns versus time in accordance with someembodiments.

In FIG. 4H, the top haptic audio pattern “Click A1 audio” is audiooutput that is played conjunction with “Click A” Normal MiniTap (230 Hz)to simulate a first down-click in a “normal” first click, as shown inFIG. 4K (first row in the First Click column) and the upper portion ofFIG. 4L, where the rate of change of intensity of a contact at a controlactivation threshold is above a threshold rate of change (e.g., thecontact is making a “normal” hard/fast press). In this example, “ClickA1 audio” is offset from the start of the “Click A” Normal MiniTap (230Hz) tactile output by 2 ms. In some cases, the same “Click A1 audio” and“Click A” Normal MiniTap (230 Hz) are played to simulate the firstup-click that follows the first down-click. In some cases, the gain ofthe “Click A1 audio” and/or “Click A” Normal MiniTap (230 Hz) arereduced (e.g., by 50%) in the up-click relative to the precedingdown-click.

The top haptic audio pattern “Click A1 audio” is also played inconjunction with “Click A” Soft MiniTap (230 Hz) to simulate a firstdown-click in a “soft” first click, as shown in FIG. 4K (second row inthe First Click column) and the lower portion of FIG. 4L, where the rateof change of intensity of a contact at a control activation threshold isbelow a threshold rate of change (e.g., the contact is making a “soft”and/or slow press). To simulate a “soft” down-click, the gain of the“Click A1 audio” and “Click A” Soft MiniTap (230 Hz) are reduced (e.g.,by 50%) in the “soft” down-click relative to the “normal” down-click. Inthis example, “Click A1 audio” is offset from the start of the “Click A”Soft MiniTap (230 Hz) tactile output by 2 ms. In some cases, the same“Click A1 audio” and “Click A” Soft MiniTap (230 Hz) are played tosimulate the first up-click that follows the first down-click. In somecases, the gain of the “Click A1 audio” and/or “Click A” Soft MiniTap(230 Hz) are reduced (e.g., by 50%) in the up-click relative to thepreceding down-click.

In FIG. 4H, the bottom haptic audio pattern “Click A2 audio” is audiooutput that is played conjunction with “Click A” Normal MiniTap (230 Hz)to simulate a second down-click in a “normal” second click that followsthe first click within a predetermined period of time (e.g., as thesecond click in a double click input), as shown in FIG. 4K (first row inthe Second Click column) and the upper portion of FIG. 4M, where therate of change of intensity of a contact at a control activationthreshold is above a threshold rate of change (e.g., the contact in thesecond click is making a “normal” hard/fast press). In this example,“Click A2 audio” is offset from the start of the “Click A” NormalMiniTap (230 Hz) tactile output by 2 ms. In some cases, the same “ClickA2 audio” and “Click A” Normal MiniTap (230 Hz) are played to simulatethe second up-click that follows the second down-click. In some cases,the gain of the “Click A2 audio” and/or “Click A” Normal MiniTap (230Hz) are reduced (e.g., by 50%) in the second up-click relative to thepreceding second down-click.

The bottom haptic audio pattern “Click A2 audio” is also played inconjunction with “Click A” Soft MiniTap (230 Hz) to simulate a seconddown-click in a “soft” second click that follows the first click withina predetermined period of time (e.g., as the second click in a doubleclick input), as shown in FIG. 4K (second row in the Second Clickcolumn) and the lower portion of FIG. 4M, where the rate of change ofintensity of a contact at a control activation threshold is below athreshold rate of change (e.g., the contact is making a “soft” and/orslow press). To simulate a “soft” down-click, the gain of the “Click A2audio” and “Click A” Soft MiniTap (230 Hz) are reduced (e.g., by 50%) inthe “soft” down-click relative to the “normal” down-click. In thisexample, “Click A2 audio” is offset from the start of the “Click A” SoftMiniTap (230 Hz) tactile output by 2 ms. In some cases, the same “ClickA2 audio” and “Click A” Soft MiniTap (230 Hz) are played to simulate thesecond up-click that follows the second down-click. In some cases, thegain of the “Click A2 audio” and/or “Click A” Soft MiniTap (230 Hz) arereduced (e.g., by 50%) in the second up-click relative to the precedingsecond down-click.

In FIG. 4I, the top haptic audio pattern “Click B1 audio” is audiooutput that is played conjunction with “Click B” Normal MiniTap (270 Hz)to simulate a first down-click in a “normal” first click, as shown inFIG. 4K (third row in the First Click column) and the upper portion ofFIG. 4N, where the rate of change of intensity of a contact at a controlactivation threshold is above a threshold rate of change (e.g., thecontact is making a “normal” hard/fast press). In this example, “ClickB1 audio” is offset from the start of the “Click B” Normal MiniTap (270Hz) tactile output by 2.8 ms. In some cases, the same “Click B1 audio”and “Click B” Normal MiniTap (270 Hz) are played to simulate the firstup-click that follows the first down-click. In some cases, the gain ofthe “Click B1 audio” and/or “Click B” Normal MiniTap (270 Hz) arereduced (e.g., by 50%) in the up-click relative to the precedingdown-click.

The top haptic audio pattern “Click B1 audio” is also played inconjunction with “Click B” Soft MiniTap (270 Hz) to simulate a firstdown-click in a “soft” first click, as shown in FIG. 4K (fourth row inthe First Click column) and the lower portion of FIG. 4N, where the rateof change of intensity of a contact at a control activation threshold isbelow a threshold rate of change (e.g., the contact is making a “soft”and/or slow press). To simulate a “soft” down-click, the gain of the“Click B1 audio” and “Click B” Soft MiniTap (270 Hz) are reduced (e.g.,by 50%) in the “soft” down-click relative to the “normal” down-click. Inthis example, “Click B1 audio” is offset from the start of the “Click B”Soft MiniTap (270 Hz) tactile output by 2.8 ms. In some cases, the same“Click B1 audio” and “Click B” Soft MiniTap (270 Hz) are played tosimulate the first up-click that follows the first down-click. In somecases, the gain of the “Click B1 audio” and/or “Click B” Soft MiniTap(230 Hz) are reduced (e.g., by 50%) in the up-click relative to thepreceding down-click.

In FIG. 4I, the bottom haptic audio pattern “Click B2 audio” is audiooutput that is played conjunction with “Click B” Normal MiniTap (270 Hz)to simulate a second down-click in a “normal” second click that followsthe first click within a predetermined period of time (e.g., as thesecond click in a double click input), as shown in FIG. 4K (third row inthe Second Click column) and the upper portion of FIG. 4O, where therate of change of intensity of a contact at a control activationthreshold is above a threshold rate of change (e.g., the contact in thesecond click is making a “normal” hard/fast press). In this example,“Click B2 audio” is offset from the start of the “Click B” NormalMiniTap (270 Hz) tactile output by 2.8 ms. In some cases, the same“Click B2 audio” and “Click B” Normal MiniTap (230 Hz) are played tosimulate the second up-click that follows the second down-click. In somecases, the gain of the “Click B2 audio” and/or “Click B” Normal MiniTap(270 Hz) are reduced (e.g., by 50%) in the second up-click relative tothe preceding second down-click.

The bottom haptic audio pattern “Click B2 audio” is also played inconjunction with “Click B” Soft MiniTap (270 Hz) to simulate a seconddown-click in a “soft” second click that follows the first click withina predetermined period of time (e.g., as the second click in a doubleclick input), as shown in FIG. 4K (fourth row in the Second Clickcolumn) and the lower portion of FIG. 4O, where the rate of change ofintensity of a contact at a control activation threshold is below athreshold rate of change (e.g., the contact is making a “soft” and/orslow press). To simulate a “soft” down-click, the gain of the “Click B2audio” and “Click B” Soft MiniTap (270 Hz) are reduced (e.g., by 50%) inthe “soft” down-click relative to the “normal” down-click. In thisexample, “Click B2 audio” is offset from the start of the “Click B” SoftMiniTap (270 Hz) tactile output by 2.8 ms. In some cases, the same“Click B2 audio” and “Click B” Soft MiniTap (270 Hz) are played tosimulate the second up-click that follows the second down-click. In somecases, the gain of the “Click B2 audio” and/or “Click B” Soft MiniTap(270 Hz) are reduced (e.g., by 50%) in the second up-click relative tothe preceding second down-click.

In FIG. 4J, the top haptic audio pattern “Click C1 audio” is audiooutput that is played conjunction with “Click C” Normal MiniTap (300 Hz)to simulate a first down-click in a “normal” first click, as shown inFIG. 4K (fifth row in the First Click column) and the upper portion ofFIG. 4P, where the rate of change of intensity of a contact at a controlactivation threshold is above a threshold rate of change (e.g., thecontact is making a “normal” hard/fast press). In this example, “ClickC1 audio” is offset from the start of the “Click C” Normal MiniTap (300Hz) tactile output by 1.9 ms. In some cases, the same “Click C1 audio”and “Click C” Normal MiniTap (300 Hz) are played to simulate the firstup-click that follows the first down-click. In some cases, the gain ofthe “Click C1 audio” and/or “Click C” Normal MiniTap (300 Hz) arereduced (e.g., by 50%) in the up-click relative to the precedingdown-click.

The top haptic audio pattern “Click C1 audio” is also played inconjunction with “Click C” Soft MiniTap (300 Hz) to simulate a firstdown-click in a “soft” first click, as shown in FIG. 4K (sixth row inthe First Click column) and the lower portion of FIG. 4P, where the rateof change of intensity of a contact at a control activation threshold isbelow a threshold rate of change (e.g., the contact is making a “soft”and/or slow press). To simulate a “soft” down-click, the gain of the“Click C1 audio” and “Click C” Soft MiniTap (300 Hz) are reduced (e.g.,by 50%) in the “soft” down-click relative to the “normal” down-click. Inthis example, “Click C1 audio” is offset from the start of the “Click C”Soft MiniTap (300 Hz) tactile output by 1.9 ms. In some cases, the same“Click C1 audio” and “Click C” Soft MiniTap (270 Hz) are played tosimulate the first up-click that follows the first down-click. In somecases, the gain of the “Click C1 audio” and/or “Click C” Soft MiniTap(300 Hz) are reduced (e.g., by 50%) in the up-click relative to thepreceding down-click.

In FIG. 4J, the bottom haptic audio pattern “Click C2 audio” is audiooutput that is played conjunction with “Click C” Normal MiniTap (300 Hz)to simulate a second down-click in a “normal” second click that followsthe first click within a predetermined period of time (e.g., as thesecond click in a double click input), as shown in FIG. 4K (fifth row inthe Second Click column) and the upper portion of FIG. 4Q, where therate of change of intensity of a contact at a control activationthreshold is above a threshold rate of change (e.g., the contact in thesecond click is making a “normal” hard/fast press). In this example,“Click C2 audio” is offset from the start of the “Click C” NormalMiniTap (300 Hz) tactile output by 1.9 ms. In some cases, the same“Click C2 audio” and “Click C” Normal MiniTap (300 Hz) are played tosimulate the second up-click that follows the second down-click. In somecases, the gain of the “Click C2 audio” and/or “Click C” Normal MiniTap(300 Hz) are reduced (e.g., by 50%) in the second up-click relative tothe preceding second down-click.

The bottom haptic audio pattern “Click C2 audio” is also played inconjunction with “Click C” Soft MiniTap (300 Hz) to simulate a seconddown-click in a “soft” second click that follows the first click withina predetermined period of time (e.g., as the second click in a doubleclick input), as shown in FIG. 4K (sixth row in the Second Click column)and the lower portion of FIG. 4Q, where the rate of change of intensityof a contact at a control activation threshold is below a threshold rateof change (e.g., the contact is making a “soft” and/or slow press). Tosimulate a “soft” down-click, the gain of the “Click C2 audio” and“Click C” Soft MiniTap (300 Hz) are reduced (e.g., by 50%) in the “soft”down-click relative to the “normal” down-click. In this example, “ClickC2 audio” is offset from the start of the “Click C” Soft MiniTap (300Hz) tactile output by 1.9 ms. In some cases, the same “Click C2 audio”and “Click C” Soft MiniTap (300 Hz) are played to simulate the secondup-click that follows the second down-click. In some cases, the gain ofthe “Click C2 audio” and/or “Click C” Soft MiniTap (300 Hz) are reduced(e.g., by 50%) in the second up-click relative to the preceding seconddown-click.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 1A areimplemented in hardware, software, firmware, or a combination thereof,including one or more signal processing and/or application specificintegrated circuits.

Memory 102 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 102 by othercomponents of device 100, such as CPU(s) 120 and the peripheralsinterface 118, is, optionally, controlled by memory controller 122.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU(s) 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data.

In some embodiments, peripherals interface 118, CPU(s) 120, and memorycontroller 122 are, optionally, implemented on a single chip, such aschip 104. In some other embodiments, they are, optionally, implementedon separate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The wirelesscommunication optionally uses any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and/or IEEE802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol fore-mail (e.g., Internet message access protocol (IMAP) and/or post officeprotocol (POP)), instant messaging (e.g., extensible messaging andpresence protocol (XMPP), Session Initiation Protocol for InstantMessaging and Presence Leveraging Extensions (SIMPLE), Instant Messagingand Presence Service (IMPS)), and/or Short Message Service (SMS), or anyother suitable communication protocol, including communication protocolsnot yet developed as of the filing date of this document.

Audio circuitry 110, speaker 111, and microphone 113 provide an audiointerface between a user and device 100. Audio circuitry 110 receivesaudio data from peripherals interface 118, converts the audio data to anelectrical signal, and transmits the electrical signal to speaker 111.Speaker 111 converts the electrical signal to human-audible sound waves.Audio circuitry 110 also receives electrical signals converted bymicrophone 113 from sound waves. Audio circuitry 110 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 118 for processing. Audio data is, optionally,retrieved from and/or transmitted to memory 102 and/or RF circuitry 108by peripherals interface 118. In some embodiments, audio circuitry 110also includes a headset jack (e.g., 212, FIG. 2A). The headset jackprovides an interface between audio circuitry 110 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch-sensitive display system 112 and other input or control devices116, with peripherals interface 118. I/O subsystem 106 optionallyincludes display controller 156, optical sensor controller 158,intensity sensor controller 159, haptic feedback controller 161, and oneor more input controllers 160 for other input or control devices. Theone or more input controllers 160 receive/send electrical signalsfrom/to other input or control devices 116. The other input or controldevices 116 optionally include physical buttons (e.g., push buttons,rocker buttons, etc.), dials, slider switches, joysticks, click wheels,and so forth. In some alternate embodiments, input controller(s) 160are, optionally, coupled with any (or none) of the following: akeyboard, infrared port, USB port, stylus, and/or a pointer device suchas a mouse. The one or more buttons (e.g., 208, FIG. 2A) optionallyinclude an up/down button for volume control of speaker 111 and/ormicrophone 113. The one or more buttons optionally include a push button(e.g., 206, FIG. 2A).

Touch-sensitive display system 112 provides an input interface and anoutput interface between the device and a user. Display controller 156receives and/or sends electrical signals from/to touch-sensitive displaysystem 112. Touch-sensitive display system 112 displays visual output tothe user. The visual output optionally includes graphics, text, icons,video, and any combination thereof (collectively termed “graphics”). Insome embodiments, some or all of the visual output corresponds to userinterface objects. As used herein, the term “affordance” is auser-interactive graphical user interface object (e.g., a graphical userinterface object that is configured to respond to inputs directed towardthe graphical user interface object). Examples of user-interactivegraphical user interface objects include, without limitation, a button,slider, icon, selectable menu item, switch, hyperlink, or other userinterface control.

Touch-sensitive display system 112 has a touch-sensitive surface, sensoror set of sensors that accepts input from the user based on hapticand/or tactile contact. Touch-sensitive display system 112 and displaycontroller 156 (along with any associated modules and/or sets ofinstructions in memory 102) detect contact (and any movement or breakingof the contact) on touch-sensitive display system 112 and converts thedetected contact into interaction with user-interface objects (e.g., oneor more soft keys, icons, web pages or images) that are displayed ontouch-sensitive display system 112. In an example embodiment, a point ofcontact between touch-sensitive display system 112 and the usercorresponds to a finger of the user or a stylus.

Touch-sensitive display system 112 optionally uses LCD (liquid crystaldisplay) technology, LPD (light emitting polymer display) technology, orLED (light emitting diode) technology, although other displaytechnologies are used in other embodiments. Touch-sensitive displaysystem 112 and display controller 156 optionally detect contact and anymovement or breaking thereof using any of a plurality of touch sensingtechnologies now known or later developed, including but not limited tocapacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with touch-sensitive displaysystem 112. In an example embodiment, projected mutual capacitancesensing technology is used, such as that found in the iPhone®, iPodTouch®, and iPad® from Apple Inc. of Cupertino, Calif.

Touch-sensitive display system 112 optionally has a video resolution inexcess of 100 dpi. In some embodiments, the touch screen videoresolution is in excess of 400 dpi (e.g., 500 dpi, 800 dpi, or greater).The user optionally makes contact with touch-sensitive display system112 using any suitable object or appendage, such as a stylus, a finger,and so forth. In some embodiments, the user interface is designed towork with finger-based contacts and gestures, which can be less precisethan stylus-based input due to the larger area of contact of a finger onthe touch screen. In some embodiments, the device translates the roughfinger-based input into a precise pointer/cursor position or command forperforming the actions desired by the user.

In some embodiments, in addition to the touch screen, device 100optionally includes a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad is, optionally, atouch-sensitive surface that is separate from touch-sensitive displaysystem 112 or an extension of the touch-sensitive surface formed by thetouch screen.

In some embodiments, in addition to the touch screen and/or touchpad,device 100 optionally include one or more off-display touch-sensitiveinput regions that are separate from the touch-screen and/or touch pad.In some embodiments, the off-display touch-sensitive input regions arepositioned on one or more peripheral-sides of the device that isadjacent to the display-side of the device. A touch-sensitive surface ofthe touch-sensitive input regions are optionally perpendicular to thedisplay of the device. In some embodiments, in addition to being able todetect different levels of intensity of a contact with the input region,the input region is also configured to detect movement in one or moredirections across the surface of the input region. In some embodiments,the device will recognize different types of inputs (e.g., press inputs,scroll inputs, and toggle inputs) on the input region based on themovement direction and intensity of the contact detected on the inputregion, and the device will generate localized tactile outputs at theinput region to simulate the haptic feedback provided by a mechanicalinput device, such as a switch, a slider, a dial, a wheel, or a button.In some embodiments, the off-display touch-sensitive input region isdistinct in nature from a touchpad in that, the location and movement ofa contact on the off-display touch-sensitive input region does notdirectly correspond to a location and movement of a focus selector onthe display in the manner that the location and movement of a contact onthe touchpad would correspond to a location and movement of a focusselector on the display. In fact, frequently, a focus selector is notdisplayed or required on the display when an off-display input region isused by a user to interact with the device and manipulate a userinterface shown on the display. In many cases, the focus of an input onthe input region is preset to a location, an object, or a function priorto detection of the input on the input region, and/or based on the typeof the input that is detected, as opposed to an exact location of acontact detected on the input region.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 optionally includes a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164.FIG. 1A shows an optical sensor coupled with optical sensor controller158 in I/O subsystem 106. Optical sensor(s) 164 optionally includecharge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. Optical sensor(s) 164 receive light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with imaging module 143(also called a camera module), optical sensor(s) 164 optionally capturestill images and/or video. In some embodiments, an optical sensor islocated on the back of device 100, opposite touch-sensitive displaysystem 112 on the front of the device, so that the touch screen isenabled for use as a viewfinder for still and/or video imageacquisition. In some embodiments, another optical sensor is located onthe front of the device so that the user's image is obtained (e.g., forselfies, for videoconferencing while the user views the other videoconference participants on the touch screen, etc.).

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 1A shows a contact intensity sensor coupled withintensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor(s) 165 optionally include one or more piezoresistive straingauges, capacitive force sensors, electric force sensors, piezoelectricforce sensors, optical force sensors, capacitive touch-sensitivesurfaces, or other intensity sensors (e.g., sensors used to measure theforce (or pressure) of a contact on a touch-sensitive surface). Contactintensity sensor(s) 165 receive contact intensity information (e.g.,pressure information or a proxy for pressure information) from theenvironment. In some embodiments, at least one contact intensity sensoris collocated with, or proximate to, a touch-sensitive surface (e.g.,touch-sensitive display system 112). In some embodiments, at least onecontact intensity sensor is located on the back of device 100, oppositetouch-screen display system 112 which is located on the front of device100.

Device 100 optionally also includes one or more proximity sensors 166.FIG. 1A shows proximity sensor 166 coupled with peripherals interface118. Alternately, proximity sensor 166 is coupled with input controller160 in I/O subsystem 106. In some embodiments, the proximity sensorturns off and disables touch-sensitive display system 112 when themultifunction device is placed near the user's ear (e.g., when the useris making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 167. FIG. 1A shows a tactile output generator coupled withhaptic feedback controller 161 in I/O subsystem 106. Tactile outputgenerator(s) 167 optionally include one or more electroacoustic devicessuch as speakers or other audio components and/or electromechanicaldevices that convert energy into linear motion such as a motor,solenoid, electroactive polymer, piezoelectric actuator, electrostaticactuator, or other tactile output generating component (e.g., acomponent that converts electrical signals into tactile outputs on thedevice). Tactile output generator(s) 167 receive tactile feedbackgeneration instructions from haptic feedback module 133 and generatestactile outputs on device 100 that are capable of being sensed by a userof device 100. In some embodiments, at least one tactile outputgenerator is collocated with, or proximate to, a touch-sensitive surface(e.g., touch-sensitive display system 112) and, optionally, generates atactile output by moving the touch-sensitive surface vertically (e.g.,in/out of a surface of device 100) or laterally (e.g., back and forth inthe same plane as a surface of device 100). In some embodiments, atleast one tactile output generator sensor is located on the back ofdevice 100, opposite touch-sensitive display system 112, which islocated on the front of device 100.

Device 100 optionally also includes one or more accelerometers 168. FIG.1A shows accelerometer 168 coupled with peripherals interface 118.Alternately, accelerometer 168 is, optionally, coupled with an inputcontroller 160 in I/O subsystem 106. In some embodiments, information isdisplayed on the touch-screen display in a portrait view or a landscapeview based on an analysis of data received from the one or moreaccelerometers. Device 100 optionally includes, in addition toaccelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASSor other global navigation system) receiver (not shown) for obtaininginformation concerning the location and orientation (e.g., portrait orlandscape) of device 100.

In some embodiments, the software components stored in memory 102include operating system 126, communication module (or set ofinstructions) 128, contact/motion module (or set of instructions) 130,graphics module (or set of instructions) 132, haptic feedback module (orset of instructions) 133, text input module (or set of instructions)134, Global Positioning System (GPS) module (or set of instructions)135, and applications (or sets of instructions) 136. Furthermore, insome embodiments, memory 102 stores device/global internal state 157, asshown in FIGS. 1A and 3. Device/global internal state 157 includes oneor more of: active application state, indicating which applications, ifany, are currently active; display state, indicating what applications,views or other information occupy various regions of touch-sensitivedisplay system 112; sensor state, including information obtained fromthe device's various sensors and other input or control devices 116; andlocation and/or positional information concerning the device's locationand/or attitude.

Operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X,WINDOWS, or an embedded operating system such as VxWorks) includesvarious software components and/or drivers for controlling and managinggeneral system tasks (e.g., memory management, storage device control,power management, etc.) and facilitates communication between varioushardware and software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used in some iPhone®, iPod Touch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. In some embodiments, the externalport is a Lightning connector that is the same as, or similar to and/orcompatible with the Lightning connector used in some iPhone®, iPodTouch®, and iPad® devices from Apple Inc. of Cupertino, Calif.

Contact/motion module 130 optionally detects contact withtouch-sensitive display system 112 (in conjunction with displaycontroller 156) and other touch-sensitive devices (e.g., a touchpad orphysical click wheel). Contact/motion module 130 includes varioussoftware components for performing various operations related todetection of contact (e.g., by a finger or by a stylus), such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact), determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts or stylus contacts) or to multiplesimultaneous contacts (e.g., “multitouch”/multiple finger contacts). Insome embodiments, contact/motion module 130 and display controller 156detect contact on a touchpad.

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and subsequentlyfollowed by detecting a finger-up (lift off) event. Similarly, tap,swipe, drag, and other gestures are optionally detected for a stylus bydetecting a particular contact pattern for the stylus.

In some embodiments, detecting a finger tap gesture depends on thelength of time between detecting the finger-down event and the finger-upevent, but is independent of the intensity of the finger contact betweendetecting the finger-down event and the finger-up event. In someembodiments, a tap gesture is detected in accordance with adetermination that the length of time between the finger-down event andthe finger-up event is less than a predetermined value (e.g., less than0.1, 0.2, 0.3, 0.4 or 0.5 seconds), independent of whether the intensityof the finger contact during the tap meets a given intensity threshold(greater than a nominal contact-detection intensity threshold), such asa light press or deep press intensity threshold. Thus, a finger tapgesture can satisfy particular input criteria that do not require thatthe characteristic intensity of a contact satisfy a given intensitythreshold in order for the particular input criteria to be met. Forclarity, the finger contact in a tap gesture typically needs to satisfya nominal contact-detection intensity threshold, below which the contactis not detected, in order for the finger-down event to be detected. Asimilar analysis applies to detecting a tap gesture by a stylus or othercontact. In cases where the device is capable of detecting a finger orstylus contact hovering over a touch sensitive surface, the nominalcontact-detection intensity threshold optionally does not correspond tophysical contact between the finger or stylus and the touch sensitivesurface.

The same concepts apply in an analogous manner to other types ofgestures. For example, a swipe gesture, a pinch gesture, a depinchgesture, and/or a long press gesture are optionally detected based onthe satisfaction of criteria that are either independent of intensitiesof contacts included in the gesture, or do not require that contact(s)that perform the gesture reach intensity thresholds in order to berecognized. For example., a swipe gesture is detected based on an amountof movement of one or more contacts; a pinch gesture is detected basedon movement of two or more contacts towards each other; a depinchgesture is detected based on movement of two or more contacts away fromeach other; and a long press gesture is detected based on a duration ofthe contact on the touch-sensitive surface with less than a thresholdamount of movement. As such, the statement that particular gesturerecognition criteria do not require that the intensity of the contact(s)meet a respective intensity threshold in order for the particulargesture recognition criteria to be met means that the particular gesturerecognition criteria are capable of being satisfied if the contact(s) inthe gesture do not reach the respective intensity threshold, and arealso capable of being satisfied in circumstances where one or more ofthe contacts in the gesture do reach or exceed the respective intensitythreshold. In some embodiments, a tap gesture is detected based on adetermination that the finger-down and finger-up event are detectedwithin a predefined time period, without regard to whether the contactis above or below the respective intensity threshold during thepredefined time period, and a swipe gesture is detected based on adetermination that the contact movement is greater than a predefinedmagnitude, even if the contact is above the respective intensitythreshold at the end of the contact movement. Even in implementationswhere detection of a gesture is influenced by the intensities ofcontacts performing the gesture (e.g., the device detects a long pressmore quickly when the intensity of the contact is above an intensitythreshold or delays detection of a tap input when the intensity of thecontact is higher), the detection of those gestures does not requirethat the contacts reach a particular intensity threshold so long as thecriteria for recognizing the gesture can be met in circumstances wherethe contact does not reach the particular intensity threshold (e.g.,even if the amount of time that it takes to recognize the gesturechanges).

Contact intensity thresholds, duration thresholds, and movementthresholds are, in some circumstances, combined in a variety ofdifferent combinations in order to create heuristics for distinguishingtwo or more different gestures directed to the same input element orregion so that multiple different interactions with the same inputelement are enabled to provide a richer set of user interactions andresponses. The statement that a particular set of gesture recognitioncriteria do not require that the intensity of the contact(s) meet arespective intensity threshold in order for the particular gesturerecognition criteria to be met does not preclude the concurrentevaluation of other intensity-dependent gesture recognition criteria toidentify other gestures that do have a criteria that is met when agesture includes a contact with an intensity above the respectiveintensity threshold. For example, in some circumstances, first gesturerecognition criteria for a first gesture—which do not require that theintensity of the contact(s) meet a respective intensity threshold inorder for the first gesture recognition criteria to be met—are incompetition with second gesture recognition criteria for a secondgesture—which are dependent on the contact(s) reaching the respectiveintensity threshold. In such competitions, the gesture is, optionally,not recognized as meeting the first gesture recognition criteria for thefirst gesture if the second gesture recognition criteria for the secondgesture are met first. For example, if a contact reaches the respectiveintensity threshold before the contact moves by a predefined amount ofmovement, a deep press gesture is detected rather than a swipe gesture.Conversely, if the contact moves by the predefined amount of movementbefore the contact reaches the respective intensity threshold, a swipegesture is detected rather than a deep press gesture. Even in suchcircumstances, the first gesture recognition criteria for the firstgesture still do not require that the intensity of the contact(s) meet arespective intensity threshold in order for the first gesturerecognition criteria to be met because if the contact stayed below therespective intensity threshold until an end of the gesture (e.g., aswipe gesture with a contact that does not increase to an intensityabove the respective intensity threshold), the gesture would have beenrecognized by the first gesture recognition criteria as a swipe gesture.As such, particular gesture recognition criteria that do not requirethat the intensity of the contact(s) meet a respective intensitythreshold in order for the particular gesture recognition criteria to bemet will (A) in some circumstances ignore the intensity of the contactwith respect to the intensity threshold (e.g. for a tap gesture) and/or(B) in some circumstances still be dependent on the intensity of thecontact with respect to the intensity threshold in the sense that theparticular gesture recognition criteria (e.g., for a long press gesture)will fail if a competing set of intensity-dependent gesture recognitioncriteria (e.g., for a deep press gesture) recognize an input ascorresponding to an intensity-dependent gesture before the particulargesture recognition criteria recognize a gesture corresponding to theinput (e.g., for a long press gesture that is competing with a deeppress gesture for recognition).

Graphics module 132 includes various known software components forrendering and displaying graphics on touch-sensitive display system 112or other display, including components for changing the visual impact(e.g., brightness, transparency, saturation, contrast or other visualproperty) of graphics that are displayed. As used herein, the term“graphics” includes any object that can be displayed to a user,including without limitation text, web pages, icons (such asuser-interface objects including soft keys), digital images, videos,animations and the like.

In some embodiments, graphics module 132 stores data representinggraphics to be used. Each graphic is, optionally, assigned acorresponding code. Graphics module 132 receives, from applicationsetc., one or more codes specifying graphics to be displayed along with,if necessary, coordinate data and other graphic property data, and thengenerates screen image data to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions used by tactile output generator(s) 167 toproduce tactile outputs at one or more locations on device 100 inresponse to user interactions with device 100.

Text input module 134, which is, optionally, a component of graphicsmodule 132, provides soft keyboards for entering text in variousapplications (e.g., contacts 137, e-mail 140, IM 141, browser 147, andany other application that needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing, to camera 143 as picture/video metadata,and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 136 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   contacts module 137 (sometimes called an address book or contact        list);    -   telephone module 138;    -   video conferencing module 139;    -   e-mail client module 140;    -   instant messaging (IM) module 141;    -   workout support module 142;    -   camera module 143 for still and/or video images;    -   image management module 144;    -   browser module 147;    -   calendar module 148;    -   widget modules 149, which optionally include one or more of:        weather widget 149-1, stocks widget 149-2, calculator widget        149-3, alarm clock widget 149-4, dictionary widget 149-5, and        other widgets obtained by the user, as well as user-created        widgets 149-6;    -   widget creator module 150 for making user-created widgets 149-6;    -   search module 151;    -   video and music player module 152, which is, optionally, made up        of a video player module and a music player module;    -   notes module 153;    -   map module 154; and/or    -   online video module 155.

Examples of other applications 136 that are, optionally, stored inmemory 102 include other word processing applications, other imageediting applications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, contacts module 137 includes executable instructions tomanage an address book or contact list (e.g., stored in applicationinternal state 192 of contacts module 137 in memory 102 or memory 370),including: adding name(s) to the address book; deleting name(s) from theaddress book; associating telephone number(s), e-mail address(es),physical address(es) or other information with a name; associating animage with a name; categorizing and sorting names; providing telephonenumbers and/or e-mail addresses to initiate and/or facilitatecommunications by telephone 138, video conference 139, e-mail 140, or IM141; and so forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, contact module 130, graphics module 132, and text input module 134,telephone module 138 includes executable instructions to enter asequence of characters corresponding to a telephone number, access oneor more telephone numbers in address book 137, modify a telephone numberthat has been entered, dial a respective telephone number, conduct aconversation and disconnect or hang up when the conversation iscompleted. As noted above, the wireless communication optionally usesany of a plurality of communications standards, protocols andtechnologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, optical sensor(s) 164, optical sensor controller 158, contactmodule 130, graphics module 132, text input module 134, contact list137, and telephone module 138, videoconferencing module 139 includesexecutable instructions to initiate, conduct, and terminate a videoconference between a user and one or more other participants inaccordance with user instructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, e-mail client module 140 includes executableinstructions to create, send, receive, and manage e-mail in response touser instructions. In conjunction with image management module 144,e-mail client module 140 makes it very easy to create and send e-mailswith still or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, the instant messaging module 141 includesexecutable instructions to enter a sequence of characters correspondingto an instant message, to modify previously entered characters, totransmit a respective instant message (for example, using a ShortMessage Service (SMS) or Multimedia Message Service (MMS) protocol fortelephony-based instant messages or using XMPP, SIMPLE, Apple PushNotification Service (APNs) or IMPS for Internet-based instantmessages), to receive instant messages, and to view received instantmessages. In some embodiments, transmitted and/or received instantmessages optionally include graphics, photos, audio files, video filesand/or other attachments as are supported in a MMS and/or an EnhancedMessaging Service (EMS). As used herein, “instant messaging” refers toboth telephony-based messages (e.g., messages sent using SMS or MMS) andInternet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs,or IMPS).

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,text input module 134, GPS module 135, map module 154, and music playermodule 146, workout support module 142 includes executable instructionsto create workouts (e.g., with time, distance, and/or calorie burninggoals); communicate with workout sensors (in sports devices and smartwatches); receive workout sensor data; calibrate sensors used to monitora workout; select and play music for a workout; and display, store andtransmit workout data.

In conjunction with touch-sensitive display system 112, displaycontroller 156, optical sensor(s) 164, optical sensor controller 158,contact module 130, graphics module 132, and image management module144, camera module 143 includes executable instructions to capture stillimages or video (including a video stream) and store them into memory102, modify characteristics of a still image or video, and/or delete astill image or video from memory 102.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, text inputmodule 134, and camera module 143, image management module 144 includesexecutable instructions to arrange, modify (e.g., edit), or otherwisemanipulate, label, delete, present (e.g., in a digital slide show oralbum), and store still and/or video images.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, and text input module 134, browser module 147 includes executableinstructions to browse the Internet in accordance with userinstructions, including searching, linking to, receiving, and displayingweb pages or portions thereof, as well as attachments and other fileslinked to web pages.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, e-mail client module 140, and browser module147, calendar module 148 includes executable instructions to create,display, modify, and store calendars and data associated with calendars(e.g., calendar entries, to do lists, etc.) in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, widget modules 149are mini-applications that are, optionally, downloaded and used by auser (e.g., weather widget 149-1, stocks widget 149-2, calculator widget149-3, alarm clock widget 149-4, and dictionary widget 149-5) or createdby the user (e.g., user-created widget 149-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, the widget creatormodule 150 includes executable instructions to create widgets (e.g.,turning a user-specified portion of a web page into a widget).

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, and text inputmodule 134, search module 151 includes executable instructions to searchfor text, music, sound, image, video, and/or other files in memory 102that match one or more search criteria (e.g., one or more user-specifiedsearch terms) in accordance with user instructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, and browser module 147, video andmusic player module 152 includes executable instructions that allow theuser to download and play back recorded music and other sound filesstored in one or more file formats, such as MP3 or AAC files, andexecutable instructions to display, present or otherwise play backvideos (e.g., on touch-sensitive display system 112, or on an externaldisplay connected wirelessly or via external port 124). In someembodiments, device 100 optionally includes the functionality of an MP3player, such as an iPod (trademark of Apple Inc.).

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, notes module 153 includes executable instructions to createand manage notes, to do lists, and the like in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, GPS module 135, and browser module 147, mapmodule 154 includes executable instructions to receive, display, modify,and store maps and data associated with maps (e.g., driving directions;data on stores and other points of interest at or near a particularlocation; and other location-based data) in accordance with userinstructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, text input module 134, e-mail clientmodule 140, and browser module 147, online video module 155 includesexecutable instructions that allow the user to access, browse, receive(e.g., by streaming and/or download), play back (e.g., on the touchscreen 112, or on an external display connected wirelessly or viaexternal port 124), send an e-mail with a link to a particular onlinevideo, and otherwise manage online videos in one or more file formats,such as H.264. In some embodiments, instant messaging module 141, ratherthan e-mail client module 140, is used to send a link to a particularonline video.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 102 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 102 optionally stores additionalmodules and data structures not described above.

In some embodiments, device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device100, the number of physical input control devices (such as push buttons,dials, and the like) on device 100 is, optionally, reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some embodiments, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such embodiments, a “menubutton” is implemented using a touchpad. In some other embodiments, themenu button is a physical push button or other physical input controldevice instead of a touchpad.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments. In some embodiments,memory 102 (in FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g.,in operating system 126) and a respective application 136-1 (e.g., anyof the aforementioned applications 136, 137-155, 380-390).

Event sorter 170 receives event information and determines theapplication 136-1 and application view 191 of application 136-1 to whichto deliver the event information. Event sorter 170 includes eventmonitor 171 and event dispatcher module 174. In some embodiments,application 136-1 includes application internal state 192, whichindicates the current application view(s) displayed on touch-sensitivedisplay system 112 when the application is active or executing. In someembodiments, device/global internal state 157 is used by event sorter170 to determine which application(s) is (are) currently active, andapplication internal state 192 is used by event sorter 170 to determineapplication views 191 to which to deliver event information.

In some embodiments, application internal state 192 includes additionalinformation, such as one or more of: resume information to be used whenapplication 136-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 136-1, a state queue for enabling the user to go back toa prior state or view of application 136-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 171 receives event information from peripherals interface118. Event information includes information about a sub-event (e.g., auser touch on touch-sensitive display system 112, as part of amulti-touch gesture). Peripherals interface 118 transmits information itreceives from I/O subsystem 106 or a sensor, such as proximity sensor166, accelerometer(s) 168, and/or microphone 113 (through audiocircuitry 110). Information that peripherals interface 118 receives fromI/O subsystem 106 includes information from touch-sensitive displaysystem 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to the peripheralsinterface 118 at predetermined intervals. In response, peripheralsinterface 118 transmits event information. In other embodiments,peripheral interface 118 transmits event information only when there isa significant event (e.g., receiving an input above a predeterminednoise threshold and/or for more than a predetermined duration).

In some embodiments, event sorter 170 also includes a hit viewdetermination module 172 and/or an active event recognizer determinationmodule 173.

Hit view determination module 172 provides software procedures fordetermining where a sub-event has taken place within one or more views,when touch-sensitive display system 112 displays more than one view.Views are made up of controls and other elements that a user can see onthe display.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected optionally correspond to programmatic levelswithin a programmatic or view hierarchy of the application. For example,the lowest level view in which a touch is detected is, optionally,called the hit view, and the set of events that are recognized as properinputs are, optionally, determined based, at least in part, on the hitview of the initial touch that begins a touch-based gesture.

Hit view determination module 172 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 172identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 173 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 173 determines that only the hit view should receive a particularsequence of sub-events. In other embodiments, active event recognizerdetermination module 173 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In other embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 180). In embodiments including activeevent recognizer determination module 173, event dispatcher module 174delivers the event information to an event recognizer determined byactive event recognizer determination module 173. In some embodiments,event dispatcher module 174 stores in an event queue the eventinformation, which is retrieved by a respective event receiver module182.

In some embodiments, operating system 126 includes event sorter 170.Alternatively, application 136-1 includes event sorter 170. In yet otherembodiments, event sorter 170 is a stand-alone module, or a part ofanother module stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of eventhandlers 190 and one or more application views 191, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 191 of the application 136-1 includes one or more event recognizers180. Typically, a respective application view 191 includes a pluralityof event recognizers 180. In other embodiments, one or more of eventrecognizers 180 are part of a separate module, such as a user interfacekit (not shown) or a higher level object from which application 136-1inherits methods and other properties. In some embodiments, a respectiveevent handler 190 includes one or more of: data updater 176, objectupdater 177, GUI updater 178, and/or event data 179 received from eventsorter 170. Event handler 190 optionally utilizes or calls data updater176, object updater 177 or GUI updater 178 to update the applicationinternal state 192. Alternatively, one or more of the application views191 includes one or more respective event handlers 190. Also, in someembodiments, one or more of data updater 176, object updater 177, andGUI updater 178 are included in a respective application view 191.

A respective event recognizer 180 receives event information (e.g.,event data 179) from event sorter 170, and identifies an event from theevent information. Event recognizer 180 includes event receiver 182 andevent comparator 184. In some embodiments, event recognizer 180 alsoincludes at least a subset of: metadata 183, and event deliveryinstructions 188 (which optionally include sub-event deliveryinstructions).

Event receiver 182 receives event information from event sorter 170. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 184 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 184 includes eventdefinitions 186. Event definitions 186 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(187-1), event 2 (187-2), and others. In some embodiments, sub-events inan event 187 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first lift-off (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second lift-off (touchend) for a predetermined phase. In another example, the definition forevent 2 (187-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay system 112, and lift-off of the touch (touch end). In someembodiments, the event also includes information for one or moreassociated event handlers 190.

In some embodiments, event definition 187 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 184 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display system 112, when a touch is detected ontouch-sensitive display system 112, event comparator 184 performs a hittest to determine which of the three user-interface objects isassociated with the touch (sub-event). If each displayed object isassociated with a respective event handler 190, the event comparatoruses the result of the hit test to determine which event handler 190should be activated. For example, event comparator 184 selects an eventhandler associated with the sub-event and the object triggering the hittest.

In some embodiments, the definition for a respective event 187 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 180 determines that the series ofsub-events do not match any of the events in event definitions 186, therespective event recognizer 180 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 180 includes metadata183 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 183 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 183 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 180 activates eventhandler 190 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 180 delivers event information associated with theevent to event handler 190. Activating an event handler 190 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 180 throws a flag associated withthe recognized event, and event handler 190 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used inapplication 136-1. For example, data updater 176 updates the telephonenumber used in contacts module 137, or stores a video file used in videoplayer module 145. In some embodiments, object updater 177 creates andupdates objects used in application 136-1. For example, object updater177 creates a new user-interface object or updates the position of auser-interface object. GUI updater 178 updates the GUI. For example, GUIupdater 178 prepares display information and sends it to graphics module132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to dataupdater 176, object updater 177, and GUI updater 178. In someembodiments, data updater 176, object updater 177, and GUI updater 178are included in a single module of a respective application 136-1 orapplication view 191. In other embodiments, they are included in two ormore software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 100 withinput-devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs;movement of the device; oral instructions; detected eye movements;biometric inputs; and/or any combination thereof are optionally utilizedas inputs corresponding to sub-events which define an event to berecognized.

FIG. 1C is a block diagram illustrating a tactile output module inaccordance with some embodiments. In some embodiments, I/O subsystem 106(e.g., haptic feedback controller 161 (FIG. 1A) and/or other inputcontroller(s) 160 (FIG. 1A)) includes at least some of the examplecomponents shown in FIG. 1C. In some embodiments, peripherals interface118 includes at least some of the example components shown in FIG. 1C.

In some embodiments, the tactile output module includes haptic feedbackmodule 133. In some embodiments, haptic feedback module 133 aggregatesand combines tactile outputs for user interface feedback from softwareapplications on the electronic device (e.g., feedback that is responsiveto user inputs that correspond to displayed user interfaces and alertsand other notifications that indicate the performance of operations oroccurrence of events in user interfaces of the electronic device).Haptic feedback module 133 includes one or more of: waveform module 123(for providing waveforms used for generating tactile outputs), mixer 125(for mixing waveforms, such as waveforms in different channels),compressor 127 (for reducing or compressing a dynamic range of thewaveforms), low-pass filter 129 (for filtering out high frequency signalcomponents in the waveforms), and thermal controller 131 (for adjustingthe waveforms in accordance with thermal conditions). In someembodiments, haptic feedback module 133 is included in haptic feedbackcontroller 161 (FIG. 1A). In some embodiments, a separate unit of hapticfeedback module 133 (or a separate implementation of haptic feedbackmodule 133) is also included in an audio controller (e.g., audiocircuitry 110, FIG. 1A) and used for generating audio signals. In someembodiments, a single haptic feedback module 133 is used for generatingaudio signals and generating waveforms for tactile outputs.

In some embodiments, haptic feedback module 133 also includes triggermodule 121 (e.g., a software application, operating system, or othersoftware module that determines a tactile output is to be generated andinitiates the process for generating the corresponding tactile output).In some embodiments, trigger module 121 generates trigger signals forinitiating generation of waveforms (e.g., by waveform module 123). Forexample, trigger module 121 generates trigger signals based on presettiming criteria. In some embodiments, trigger module 121 receivestrigger signals from outside haptic feedback module 133 (e.g., in someembodiments, haptic feedback module 133 receives trigger signals fromhardware input processing module 199 located outside haptic feedbackmodule 133) and relays the trigger signals to other components withinhaptic feedback module 133 (e.g., waveform module 123) or softwareapplications that trigger operations (e.g., with trigger module 121)based on activation of the hardware input device (e.g., a home button, aside button, a power button, etc.). In some embodiments, trigger module121 also receives tactile feedback generation instructions (e.g., fromhaptic feedback module 133, FIGS. 1A and 3). In some embodiments,trigger module 121 generates trigger signals in response to hapticfeedback module 133 (or trigger module 121 in haptic feedback module133) receiving tactile feedback instructions (e.g., from haptic feedbackmodule 133, FIGS. 1A and 3).

Waveform module 123 receives trigger signals (e.g., from trigger module121) as an input, and in response to receiving trigger signals, provideswaveforms for generation of one or more tactile outputs (e.g., waveformsselected from a predefined set of waveforms designated for use bywaveform module 123, such as the waveforms described in greater detailbelow with reference to FIGS. 4F1-4G4).

Mixer 125 receives waveforms (e.g., from waveform module 123) as aninput, and mixes together the waveforms. For example, when mixer 125receives two or more waveforms (e.g., a first waveform in a firstchannel and a second waveform that at least partially overlaps with thefirst waveform in a second channel) mixer 125 outputs a combinedwaveform that corresponds to a sum of the two or more waveforms. In someembodiments, mixer 125 also modifies one or more waveforms of the two ormore waveforms to emphasize particular waveform(s) over the rest of thetwo or more waveforms (e.g., by increasing a scale of the particularwaveform(s) and/or decreasing a scale of the rest of the waveforms). Insome circumstances, mixer 125 selects one or more waveforms to removefrom the combined waveform (e.g., the waveform from the oldest source isdropped when there are waveforms from more than three sources that havebeen requested to be output concurrently by tactile output generator167)

Compressor 127 receives waveforms (e.g., a combined waveform from mixer125) as an input, and modifies the waveforms. In some embodiments,compressor 127 reduces the waveforms (e.g., in accordance with physicalspecifications of tactile output generators 167 (FIG. 1A) or 357 (FIG.3)) so that tactile outputs corresponding to the waveforms are reduced.In some embodiments, compressor 127 limits the waveforms, such as byenforcing a predefined maximum amplitude for the waveforms. For example,compressor 127 reduces amplitudes of portions of waveforms that exceed apredefined amplitude threshold while maintaining amplitudes of portionsof waveforms that do not exceed the predefined amplitude threshold. Insome embodiments, compressor 127 reduces a dynamic range of thewaveforms. In some embodiments, compressor 127 dynamically reduces thedynamic range of the waveforms so that the combined waveforms remainwithin performance specifications of the tactile output generator 167(e.g., force and/or moveable mass displacement limits).

Low-pass filter 129 receives waveforms (e.g., compressed waveforms fromcompressor 127) as an input, and filters (e.g., smooths) the waveforms(e.g., removes or reduces high frequency signal components in thewaveforms). For example, in some instances, compressor 127 includes, incompressed waveforms, extraneous signals (e.g., high frequency signalcomponents) that interfere with the generation of tactile outputs and/orexceed performance specifications of tactile output generator 167 whenthe tactile outputs are generated in accordance with the compressedwaveforms. Low-pass filter 129 reduces or removes such extraneoussignals in the waveforms.

Thermal controller 131 receives waveforms (e.g., filtered waveforms fromlow-pass filter 129) as an input, and adjusts the waveforms inaccordance with thermal conditions of device 100 (e.g., based oninternal temperatures detected within device 100, such as thetemperature of haptic feedback controller 161, and/or externaltemperatures detected by device 100). For example, in some cases, theoutput of haptic feedback controller 161 varies depending on thetemperature (e.g. haptic feedback controller 161, in response toreceiving same waveforms, generates a first tactile output when hapticfeedback controller 161 is at a first temperature and generates a secondtactile output when haptic feedback controller 161 is at a secondtemperature that is distinct from the first temperature). For example,the magnitude (or the amplitude) of the tactile outputs may varydepending on the temperature. To reduce the effect of the temperaturevariations, the waveforms are modified (e.g., an amplitude of thewaveforms is increased or decreased based on the temperature).

In some embodiments, haptic feedback module 133 (e.g., trigger module121) is coupled to hardware input processing module 199. In someembodiments, other input controller(s) 160 in FIG. 1A includes hardwareinput processing module 199. In some embodiments, hardware inputprocessing module 199 receives inputs from hardware input device 198(e.g., other input or control devices 116 in FIG. 1A, such as a homebutton, a side button, or a power button). In some embodiments, hardwareinput device 198 is any input device described herein, such astouch-sensitive display system 112 (FIG. 1A), keyboard/mouse 350 (FIG.3), touchpad 355 (FIG. 3), one of other input or control devices 116(FIG. 1A), or an intensity-sensitive button (e.g., a home button asshown in FIG. 2B, a side button on the side of the device (e.g., inputregions 455, 456, and 457) as shown in FIG. 4A), or a button with amechanical actuator as illustrated in FIG. 2C). In some embodiments,hardware input device 198 consists of an intensity-sensitive inputregion (e.g., a home button as shown in FIG. 2B, a side button on theside of the device adjacent to the display-side as illustrated in FIG.4A, or a button with a mechanical actuator as illustrated in FIG. 2C),and not touch-sensitive display system 112 (FIG. 1A), keyboard/mouse 350(FIG. 3), or touchpad 355 (FIG. 3). In some embodiments, in response toinputs from hardware input device 198, hardware input processing module199 provides one or more trigger signals to haptic feedback module 133to indicate that a user input satisfying predefined input criteria, suchas an input corresponding to a “click” of a button (e.g., a “down click”or an “up click”), has been detected. In some embodiments, hapticfeedback module 133 provides waveforms that correspond to the “click” ofa button in response to the input corresponding to the “click” of abutton, simulating a haptic feedback of pressing a physical button. Insome embodiments, haptic feedback module 133 provides waveforms thatcorrespond to the “detents” of a scroll wheel or dial in response to theinput corresponding to the “scrolling” of a scroll wheel or “turning” ofa dial, simulating a haptic feedback of rotating a physical wheel ordial. In some embodiments, haptic feedback module 133 provides waveformsthat correspond to the “snap” of a switch in response to the inputcorresponding to the “opening” or “closing” of a switch, simulating ahaptic feedback of flipping a physical switch.

In some embodiments, the tactile output module includes haptic feedbackcontroller 161 (e.g., haptic feedback controller 161 in FIG. 1A), whichcontrols the generation of tactile outputs. In some embodiments, hapticfeedback controller 161 is coupled to a plurality of tactile outputgenerators, and selects one or more tactile output generators of theplurality of tactile output generators and sends waveforms to theselected one or more tactile output generators for generating tactileoutputs. In some embodiments, haptic feedback controller 161 coordinatestactile output requests that correspond to activation of hardware inputdevice 198 and tactile output requests that correspond to softwareevents (e.g., tactile output requests from haptic feedback module 133)and modifies one or more waveforms of the two or more waveforms toemphasize particular waveform(s) over the rest of the two or morewaveforms (e.g., by increasing a scale of the particular waveform(s)and/or decreasing a scale of the rest of the waveforms, such as toprioritize tactile outputs that correspond to activations of hardwareinput device 198 over tactile outputs that correspond to softwareevents).

In some embodiments, as shown in FIG. 1C, an output of haptic feedbackcontroller 161 is coupled to audio circuitry of device 100 (e.g., audiocircuitry 110, FIG. 1A), and provides audio signals to audio circuitryof device 100. In some embodiments, haptic feedback controller 161provides both waveforms used for generating tactile outputs and audiosignals used for providing audio outputs in conjunction with generationof the tactile outputs. In some embodiments, haptic feedback controller161 modifies audio signals and/or waveforms (used for generating tactileoutputs) so that the audio outputs and the tactile outputs aresynchronized (e.g., by delaying the audio signals and/or waveforms). Insome embodiments, haptic feedback controller 161 includes adigital-to-analog converter used for converting digital waveforms intoanalog signals, which are received by amplifier 163 and/or tactileoutput generator 167.

In some embodiments, the tactile output module includes amplifier 163.In some embodiments, amplifier 163 receives waveforms (e.g., from hapticfeedback controller 161) and amplifies the waveforms prior to sendingthe amplified waveforms to tactile output generator(s) 167 (e.g., anyone or more of tactile output generators 167 (FIG. 1A) or 357 (FIG. 3)).For example, amplifier 163 amplifies the received waveforms to signallevels that are in accordance with physical specifications of tactileoutput generator(s) 167 (e.g., to a voltage and/or a current required bytactile output generator(s) 167 for generating tactile outputs so thatthe signals sent to tactile output generator(s) 167 produce tactileoutputs that correspond to the waveforms received from haptic feedbackcontroller 161) and sends the amplified waveforms to tactile outputgenerator(s) 167. In response, tactile output generator(s) 167 generatestactile outputs (e.g., localized tactile outputs and/or whole devicetactile outputs). In some embodiments, localized tactile outputs aregenerated by a surface oscillator that rocks back and forth around afixed pivot underneath the input region (e.g., a side button) or othertactile output generator that is capable of generating tactile outputsthat are directed specifically toward the input region. In someembodiments, the whole device tactile outputs are generated by shiftinga moveable mass back and forth in one or more dimensions relative to aneutral position of the moveable mass. The whole device tactile outputsare created in many different locations on the device at the same time,e.g., by a tactile output generator that is coupled to a housing of thedevice.

In some embodiments, the tactile output module includes sensor 169,which is coupled to tactile output generator 167. Sensor 169 detectsstates or state changes (e.g., mechanical position, physicaldisplacement, and/or movement) of tactile output generator 167 or one ormore components of tactile output generator 167 (e.g., one or moremoving parts, such as a membrane, used to generate tactile outputs). Insome embodiments, sensor 169 is a magnetic field sensor (e.g., a Halleffect sensor) or other displacement and/or movement sensor. In someembodiments, sensor 169 provides information (e.g., a position, adisplacement, and/or a movement of one or more parts in tactile outputgenerator 167) to haptic feedback controller 161 and, in accordance withthe information provided by sensor 169 about the state of tactile outputgenerator 167, haptic feedback controller 161 adjusts the waveformsoutput from haptic feedback controller 161 (e.g., waveforms sent totactile output generator 167, optionally via amplifier 163).

FIG. 2A illustrates a portable multifunction device 100 having a touchscreen (e.g., touch-sensitive display system 112, FIG. 1A) in accordancewith some embodiments. The touch screen optionally displays one or moregraphics within user interface (UI) 200. In this embodiment, as well asothers described below, a user is enabled to select one or more of thegraphics by making a gesture on the graphics, for example, with one ormore fingers 202 (not drawn to scale in the figure) or one or morestyluses 203 (not drawn to scale in the figure). In some embodiments,selection of one or more graphics occurs when the user breaks contactwith the one or more graphics. In some embodiments, the gestureoptionally includes one or more taps, one or more swipes (from left toright, right to left, upward and/or downward) and/or a rolling of afinger (from right to left, left to right, upward and/or downward) thathas made contact with device 100. In some implementations orcircumstances, inadvertent contact with a graphic does not select thegraphic. For example, a swipe gesture that sweeps over an applicationicon optionally does not select the corresponding application when thegesture corresponding to selection is a tap.

Device 100 optionally also includes one or more physical buttons, suchas “home” or menu button 204. As described previously, menu button 204is, optionally, used to navigate to any application 136 in a set ofapplications that are, optionally executed on device 100. Alternatively,in some embodiments, the menu button is implemented as a soft key in aGUI displayed on the touch-screen display. In some embodiments, the homebutton is removed and its functions are invoked by a predefined gestureon the touch-sensitive display (e.g., an upward swipe gesture from thebottom edge of the touch-screen display).

In some embodiments, device 100 includes the touch-screen display, menubutton 204, push button 206 for powering the device on/off and lockingthe device, volume adjustment button(s) 208, Subscriber Identity Module(SIM) card slot 210, head set jack 212, and docking/charging externalport 124. Push button 206 is, optionally, used to turn the power on/offon the device by depressing the button and holding the button in thedepressed state for a predefined time interval; to lock the device bydepressing the button and releasing the button before the predefinedtime interval has elapsed; and/or to unlock the device or initiate anunlock process. In some embodiments, device 100 also accepts verbalinput for activation or deactivation of some functions throughmicrophone 113. Device 100 also, optionally, includes one or morecontact intensity sensors 165 for detecting intensities of contacts ontouch-sensitive display system 112 and/or one or more tactile outputgenerators 167 for generating tactile outputs for a user of device 100.In some embodiments, the volume buttons 208 and the push button 206 arereplaced by an intensity-sensitive input region that is coupled withlocalized tactile output generators that provide tactile outputs forsimulating the feel of a physical button, wheel, switch or otherhardware input devices in response to different types of touch inputs.

FIGS. 2B-2C show exploded views of a first input device suitable for usein the electronic devices shown in FIGS. 1A, 2A, 3, and/or 4A (e.g., ashome button 204). FIG. 2B shows an example of an intensity-sensitivehome button with capacitive sensors used to determine a range ofintensity values that correspond to force applied to theintensity-sensitive home button. FIG. 2C shows an example of a homebutton with a mechanical switch element. With reference to FIG. 2B, theinput device stack 220 includes a cover element 222 and a trim 224. Inthe illustrated embodiment, the trim 224 completely surrounds the sidesof the cover element 222 and the perimeter of the top surface of thecover element 222. Other embodiments are not limited to thisconfiguration. For example, in one embodiment the sides and/or topsurface of the cover element 222 can be partially surrounded by the trim224. Alternatively, the trim 224 can be omitted in other embodiments.

Both the cover element 222 and the trim 224 can be formed with anysuitable opaque, transparent, and/or translucent material. For example,the cover element 222 can be made of glass, plastic, or sapphire and thetrim 224 may be made of a metal or plastic. In some embodiments, one ormore additional layers (not shown) can be positioned below the coverelement 222. For example, an opaque ink layer can be disposed below thecover element 222 when the cover element 222 is made of a transparentmaterial. The opaque ink layer can conceal the other components in theinput device stack 220 so that the other components are not visiblethrough the transparent cover element 222.

A first circuit layer 226 can be disposed below the cover element 222.Any suitable circuit layer may be used. For example, the first circuitlayer 226 may be a circuit board or a flexible circuit. The firstcircuit layer 226 can include one or more circuits, signal lines, and/orintegrated circuits. In one embodiment, the first circuit layer 226includes a biometric sensor 228. Any suitable type of biometric sensorcan be used. For example, in one embodiment the biometric sensor is acapacitive fingerprint sensor that captures at least one fingerprintwhen a user's finger (or fingers) approaches and/or contacts the coverelement 222.

The first circuit layer 226 may be attached to the bottom surface of thecover element 222 with an adhesive layer 230. Any suitable adhesive canbe used for the adhesive layer. For example, a pressure sensitiveadhesive layer may be used as the adhesive layer 230.

A compliant layer 232 is disposed below the first circuit layer 226. Inone embodiment, the compliant layer 232 includes an opening 234 formedin the compliant layer 232. The opening 234 exposes the top surface ofthe first circuit layer 226 and/or the biometric sensor 228 when thedevice stack 220 is assembled. In the illustrated embodiment, thecompliant layer 232 is positioned around an interior perimeter of thetrim 224 and/or around a peripheral edge of the cover element 222.Although depicted in a circular shape, the compliant layer 232 can haveany given shape and/or dimensions, such as a square or oval. Thecompliant layer 232 is shown as a continuous compliant layer in FIGS. 2Band 2C, but other embodiments are not limited to this configuration. Insome embodiments, multiple discrete compliant layers may be used in thedevice stack 220. Additionally, in some embodiments, the compliant layer232 does not include the opening 234 and the compliant layer 232 extendsacross at least a portion of the input device stack 220. For example,the compliant layer 232 may extend across the bottom surface of thecover element 222, the bottom surface of the first circuit layer 226, ora portion of the bottom surface of the cover element 222 (e.g., aroundthe peripheral edge of the cover element) and the bottom surface of thefirst circuit layer 226.

A second circuit layer 238 is positioned below the first circuit layer226. A flexible circuit and a circuit board are examples of a circuitlayer that can be used in the second circuit layer 238. In someembodiments, the second circuit layer 238 can include a first circuitsection 240 and a second circuit section 242. The first and secondcircuit sections 240, 242 can be electrically connected one anotherother.

The first circuit section 240 can include a first set of one or moreintensity sensor components that are included in an intensity sensor. Insome embodiments, the first circuit section 240 can be electricallyconnected to the first circuit layer 226. For example, when the firstcircuit layer 226 includes a biometric sensor 228, the biometric sensor228 may be electrically connected to the first circuit section 240 ofthe second circuit layer 238.

The second circuit section 242 can include additional circuitry, such assignal lines, circuit components, integrated circuits, and the like. Inone embodiment, the second circuit section 242 may include aboard-to-board connector 244 to electrically connect the second circuitlayer 238 to other circuitry in the electronic device. For example, thesecond circuit layer 238 can be operably connected to a processingdevice using the board-to-board connector 244. Additionally, oralternatively, the second circuit layer 238 may be operably connected tocircuitry that transmits signals (e.g., sense signals) received from theintensity sensor component(s) in the first circuit section 240 to aprocessing device. Additionally, or alternatively, the second circuitlayer 238 may be operably connected to circuitry that provides signals(e.g., drive signals, a reference signal) to the one or more intensitysensor components in the first circuit section 240.

In some embodiments, the first circuit section 240 of the second circuitlayer 238 may be attached to the bottom surface of the first circuitlayer 226 using an adhesive layer 236. In a non-limiting example, a dieattach film may be used to attach the first circuit section 240 to thebottom surface of the first circuit layer 226.

A third circuit layer 246 is disposed below the first circuit section240 of the second circuit layer 238. The third circuit layer 246 mayinclude a second set of one or more intensity sensor components that areincluded in an intensity sensor. The third circuit layer 246 issupported by and/or attached to a support element 248. In oneembodiment, the support element 248 is attached to the trim 224 toproduce an enclosure for the other components in the device stack 220.The support element 248 may be attached to the trim 224 using anysuitable attachment mechanism.

The first set of one or more intensity sensor components in the firstcircuit section 240 and the second set of one or more intensity sensorcomponents in the third circuit layer 246 together form an intensitysensor. The intensity sensor can use any suitable intensity sensingtechnology. Example sensing technologies include, but are not limitedto, capacitive, piezoelectric, piezoresistive, ultrasonic, and magnetic.

In the examples shown in FIGS. 2B and 2C, the intensity sensor is acapacitive force sensor. With a capacitive force sensor, the first setof one or more intensity sensor components can include a first set ofone or more electrodes 250 and the second set of one or more forcesensor components a second set of one or more electrodes 252. Althoughshown in a square shape in FIGS. 2B and 2C each electrode in the firstand second sets of one or more electrodes 250, 252 can have any givenshape (e.g., rectangles, circles). Additionally, the one or moreelectrodes in the first and second sets 250, 252 may be arranged in anygiven pattern (e.g., one or more rows and one or more columns).

FIG. 2B and 2C show two electrodes in the first and second sets of oneor more electrodes 250, 252. However, other embodiments are not limitedto this configuration. The first and second sets of one or moreelectrodes 250, 252 may each be a single electrode or multiple discreteelectrodes. For example, if the first set of one or more electrodes is asingle electrode, the second set of one or more electrodes comprisesmultiple discrete electrodes. In some embodiments, the second set of oneor more electrodes can be a single electrode and the first set includesmultiple discrete electrodes. Alternatively, both the first and secondsets of one or more electrodes may each include multiple discreteelectrodes.

Each electrode in the first set of one or more electrodes 250 is alignedin at least one direction (e.g., vertically) with a respective electrodein the second set of one or more electrodes 252 to produce one or morecapacitors. When a force input is applied to the cover element 222(e.g., the input surface of the input device), at least one electrode inthe first set 250 moves closer to a respective electrode in the secondset 252, which varies the capacitance of the capacitor(s). A capacitancesignal sensed from each capacitor represents a capacitance measurementof that capacitor. A processing device (not shown) is configured toreceive the capacitance signal(s) and correlate the capacitancesignal(s) to an amount of intensity applied to the cover element 222. Insome embodiments, the force sensor can replace a switch element anddifferent intensity thresholds can be used to determine activationevents.

In some embodiments, such as the embodiment shown in FIG. 2C, a switchelement 254 can be positioned below the support element 248. The switchelement 254 registers a user input when a force input applied to thecover element 222 exceeds a given amount of force (e.g., a forcethreshold associated with closing the distance between the first circuitsection 240 and the third circuit layer 246). Any suitable switchelement can be used. For example, the switch element 254 may be a domeswitch that collapses when the force input applied to the cover element222 exceeds the force threshold. When collapsed, the dome switchcompletes a circuit that is detected by a processing device andrecognized as a user input (e.g., a selection of an icon, function, orapplication). In one embodiment, the dome switch is arranged such thatthe apex of the collapsible dome is proximate to the bottom surface ofthe support plate 248. In another embodiment, the base of thecollapsible dome can be proximate to the bottom surface of the supportplate 248.

FIGS. 2D-2F show different views of an intensity-sensitive input device701 suitable for use in the electronic devices shown in FIGS. 1A, 2A, 3,and/or 4A (e.g., as input regions 255, 256, and 257). In someembodiments, the intensity-sensitive input device 701 is a pivotinginput device that implemented as a button 710. An input surface 712 ofbutton 710 optionally serves as one or more individual input regionsand/or sub-regions located on one or more peripheral sides (e.g., sidesthat are adjacent to the display-side) of the electronic device (e.g.,device 100 and 300), in accordance with some embodiments.

Button 710 includes permanent magnet 726 attached to a lower surface ofbutton 710. The permanent magnet 726 may be rigidly attached to button710, such that permanent magnet 726 moves or displaces in concert withbutton 710.

Button 710 rotates relative to body 730 about a pair of pivots 727 (or,in some embodiments, a single pivot). The pair of pivots 727 areattached to body 730 and positioned on each of first and secondprojections 733, 737, both of which are part of upper body 732. Each ofthe two pivots 727 are positioned between a respective projection 733,737 and a lower surface of button 710. Thus, the pair of pivots 727 arebelow button 710 (or other input structure) and its input surface 712.More specifically, one pivot 727 is positioned between the firstprojection 733 and a first lower end of the button, and another pivot727 is positioned between the second projection 737 and a second lowerend of the button 710. One pivot 727 is disposed on the first projection733 and another pivot 727 is disposed on the second projection 737. Thepivot(s) 727 may rotate with respect to body 730 or may be stationarywhile permitting button 710 to rotate relative to the body 730.

Input surface 712 of the button/input structure 710 may be touched,pressed, swiped in one or more directions, or otherwise interacted withby a user. In some embodiments, input surface 710 may translate,deflect, bend, or otherwise move a relatively small distance in responseto user input and/or in response to a movement of permanent magnet 726.In other embodiments, input surface 712 does not translate, deflect,bend, or otherwise move in response to a user input. Input may bedetected through various intensity sensors (e.g., force sensors, touchsensors, or combination of the two). Such sensors are not shown forsimplicity's sake.

Button 710 may include one or more steps or shelves. The one or moreshelves may aid in fitting the button to a host electronic device, suchas fitting the button 710 within an opening along an exterior of a hostelectronic device. The one or more shelves may receive a gasket, thegasket engaging one or more shelves.

With attention to FIGS. 7A-7B, button 710, in order from an upperportion (e.g., a portion extending from or facing an exterior of a hostelectronic device) to a lower portion (e.g., a portion extending into aninterior of a host electronic device), includes an input surface 712,first upper portion 713, collar 716, first shelf 718, and second shelf720. Collar 716 is narrower and/or thinner than first upper portion 713of button 710. Collar 716 and first upper portion 713 of button 710 mayhave the same general shape or may be of different shapes. As oneexample, both may be oblong (e.g., lozenge-shaped). Collar 716 ispositioned above or otherwise disposed on first shelf 718. Generally,upper button portion 713, collar 716, first shelf 718, and second shelf720 may all be formed integrally with one another or may be formedseparately and affixed to one another.

First shelf 718 of button 710 is typically wider and/or longer thancollar 716. In some embodiments, first shelf 718 is of similar oridentical width to first upper portion 713. First shelf 718 may have thesame shape as either or both of upper portion 713 and collar 716, or mayhave a different shape.

Second shelf 720 of button 710 is generally wider and/or longer thanfirst shelf 718. The second shelf 720 may have a similar shape as one ormore of first upper portion 713, collar 716, and first shelf 718 ofbutton 710, or may be differently-shaped. Second shelf 720 is positionedbelow first shelf 718. Second shelf 720 is positioned between firstshelf 718 and connector 722.

Generally, the first shelf, collar, and second shelf cooperate to definea grove, annulus, or the like extending around a perimeter of button710. A gasket or other seal may be seated in this groove.

Connector 722 is positioned below second shelf 720 and connects button710 to permanent magnet 726. Connector 722 may be positioned at acentral portion of the upper surface of permanent magnet 726. Connector722 may be connected to permanent magnet 726 along substantially all ofthe length of an upper surface of permanent magnet 726. The permanentmagnet may be rigidly connected to button 710 by way of connector 722.In some embodiments, permanent magnet 726 extends into a space withinbody 730. That is, the body may be hollow or may have multipleprojections defining a space receiving at least part of permanent magnet730.

Body 730 includes upper body 732 and lower body 736. Each of firstprojection 733 and second projection 737 are part of upper body 732; thefirst and second projections define a volume or space therebetween inwhich part of permanent magnet 726 rests. Upper body 732 and lower body736 are separated by a region of reduced width configured to receiveelectromagnet 770. The body may be attached to an enclosure of theelectronic device, or a structure within the enclosure.

Electromagnet 770 is configured to attach to body 730 and positionedrelative to permanent magnet 726 such that a magnetic field generated byelectromagnet 770 is received by permanent magnet 726 sufficient todisplace or move permanent magnet 726. Electromagnet 770 encircles body730 (specifically, the first and second projections 733, 737 and ispositioned between upper body 732 and lower body 736. More specifically,electromagnet 770 is positioned to fit around a region of reduced widthformed between upper body 732 and lower body 736. Electromagnet 770 hassidewall 776. Generally, electromagnet 770 is located below button 710(or other input structure) and its input surface 712.

The positioning of permanent magnet 726 relative to electromagnet 770modifies the operation (e.g., actuation) of button 710. Morespecifically, the magnetic interaction between electromagnet 770 andpermanent magnet 726 is influenced by the relative positioning ofpermanent magnet 726 with respect to the electromagnet. In theembodiment of FIGS. 7A-7C, lower surface 728 of permanent magnet 726 ispositioned between ends of electromagnet 770. Stated another way, ahorizontal plane extending from the lower surface 728 of permanentmagnet 726 intersects a sidewall of electromagnet 770. In oneembodiment, the horizontal plane extending from lower surface 728 ofpermanent magnet 726 intersects sidewall 776 of electromagnet 770 at amidpoint of sidewall 776 (e.g., the end of permanent magnet 726 iscoplanar with a midpoint of sidewall 776). Thus, permanent magnet 726extends halfway through the electromagnet 770.

In some embodiments, permanent magnet 726 may be replaced by a secondelectromagnet, or may be supplemented by a second electromagnet. Usingan electromagnet in place of, or in addition to, permanent magnet 726may facilitate fine control of the magnetic force exerted on button 710(or other input structure), thereby likewise providing fine control ofthe force of the haptic output. It should be appreciated that hapticoutput via button 710 or other input structure may be increased byincreasing the field strength of the second electromagnet or decreasedby decreasing its field strength. Likewise, field strength of firstelectromagnet 770 may be varied to vary haptic output force even whenpermanent magnet 726 is used instead of a second electromagnet.

Further, it should be noted that such variations in field strengthgenerally vary haptic output strength, but not travel; a distancetraveled by the input structure (e.g., button 710) and associated inputsurface varies with the distance of the pivot point from the inputsurface, as discussed above. Increases in both haptic output strengthand travel distance may increase saliency and/or perceptibility of ahaptic output.

With attention to FIGS. 7B-7C, button 710, and attached permanent magnet726, are depicted in a neutral, unactuated first button position (asshown in FIG. 7B, which is a cross-section taken along line 7A-7A ofFIG. 7A) and in an actuated second button position (FIG. 7C). Button 710actuates or pivots between the first button position and the secondbutton position through reaction of permanent magnet 728 to a magneticfield generated by electromagnet 770.

Button 710 is attached to permanent magnet 726 by way of connector 722;in many embodiments, the ends of connector 722 define pivots 727. Button710 is configured to pivot on body 730 by way of the pair of pivots 727.In other embodiments, the connector and pivot(s) may be separateelements. Body 730, permanent magnet 726, and electromagnet 770 may bedisposed within a host electronic device.

Electromagnet 770, which may encircle at least a portion of permanentmagnet 726, generates a magnetic field which interacts with permanentmagnet 726, in tum pivoting button 710 between a neutral, unactuatedfirst button position and an actuated, second button position.

When no electric current is flowing through the wire windings ofelectromagnet 770, no magnetic field is generated by electromagnet 770and permanent magnet 726 is in a neutral position that is approximatelyin the middle of the electromagnet's interior volume 777, as shown inFIG. 7B, with one end within the interior volume. This corresponds to aneutral, unactuated first button position. However, when electromagnet770 is turned on, the resulting magnetic field moves (e.g., tilts)permanent magnet 726 within interior volume 777. More specifically,permanent magnet 726 tilts or rotates about pivot(s) 723, 727 such thatits end 728 moves closer to one side of electromagnet 777, as shown inFIG.7C. Since button 710 is attached to permanent magnet 726, it alsomoves about the pivot(s) in a direction opposite the motion of thepermanent magnet. Put another way, the button (or other input surface)and permanent magnet both rotate in the same direction (e.g., clockwiseor counterclockwise) but move in opposite directions, since they arepositioned on opposing sides of the pivot(s). Thus, when theelectromagnet is activated, button 710 moves into an actuated position.This motion may provide haptic feedback to a person touching the button710 (and typically, the button's input surface 712) to indicate theinput device 701 has been actuated. In some embodiments, the button 710and permanent magnet 726 may oscillate back and forth about the pivot727 to provide haptic feedback. In some embodiments, a haptic feedbackcontroller (e.g., haptic feedback controller 161 in FIG. 1C) varies theoperating parameters of the electromagnet 777, in accordance with therequirements of a selected tactile output profile (e.g., selected basedon the characteristics of the input, and the function that is triggered,and the user interface feedback that is displayed).

The permanent magnet 726, when influenced by the magnetic field, movesfrom its neutral position (as shown in FIG. 7B) to its actuatedposition, as shown in FIG. 7C. In its neutral position, the permanentmagnet's 726 centerline is generally aligned with a major axis of theinput device 701, as is a centerline of button 710. In the actuatedposition shown in FIG. 7C, the centerline of button 712 and permanentmagnet 726 is offset from major axis 739 of input device 701 by angle711. The angle between centerline of the button 712 and major axis 739is generally the same as the angle between the centerline of permanentmagnet 726 and major axis 739.

The button/permanent magnet centerline 719 and major axis 739 intersectat pivot point 725. Pivot point 725 is positioned at the bottom of pivot727; pivot 727 is not visible in FIGS. 7B-7C but is shown in FIG. 7A.Permanent magnet 726, and thus button 710, rotates about the pivot pointin a plane defined by major axis 739 and minor axis 771 of the inputdevice 701. Typically, although not necessarily, major axis 739 passesthrough input surface 712 and button 710, while minor axis 771 isparallel to the input surface and button. Likewise, the pivot axis(which passes through pivot point 725) is generally parallel to theinput surface.

The direction of rotation about pivot point 725 may change with thedirection of current passing through electromagnet 770; thus, button 710and permanent magnet 726 may both rotate in two directions (e.g.,clockwise or counterclockwise about the pivot point 725). As previouslymentioned, permanent magnet 726 and button 710 generally move inopposite directions while rotating about the pivot point 725 and anyassociated pivot(s) 727.

A user receives haptic feedback from button 710 actuation in that theinput surface 712 of button 710 pivots with button 710. A user touchinginput surface 712 may sense the pivoting or actuation of button 710.Furthermore, the user may be able to see the pivoting of button 710 fromthe neutral, unactuated first position to the actuated, second position.

Button 710 (or other input structure) may include intensity sensors(e.g., force sensor 717) below input surface 712 and within upperportion 713; the force sensor is shown in FIG. 7B, although it should beappreciated that the location of the intensity sensors (e.g., forcesensor 717) may vary in various embodiments. For example, force sensor717 may be positioned below input surface 712 and upper portion 713instead of within the upper portion, or may be positioned below or tothe side of permanent magnet 726, or anywhere else within the inputdevice (or on a portion of an associated electronic device's enclosure).Force sensor 717 senses an input force on input surface 712 (e.g.,detecting contact of a user's finger on input surface 712) and producesan output signal. Force sensor 717 may be any type of force sensor 717known to those skilled in the art, such as a strain gauge, a capacitivesensor, a resistive sensor, an optical sensor, and so on. If forcesensor 717 is a capacitive sensor, for example, changes in capacitancemay be sensed by sensor 717 and output as an electrical output signal tothe processor. In one embodiment, the force sensor is a strain gauge.The output signal produced by force sensor 717 is received by aprocessor.

The output signal generated by force sensor 717 allows the processor(e.g., via the tactile output controller 161 in FIG. 1C) to control, forexample, electromagnet 770 (or other actuator) to effect actuation ofbutton 710 and may also be used as a system input to the electronicdevice. For example, the force sensor output may be used to indicatethat a user has pressed or otherwise interacted with (e.g., touched,swiped) the button 710 and thus control or change some function of theelectronic device.

The processor also may control any of several inputs to electromagnet770 to vary the magnetic field generated by electromagnet 770. Forexample, the processor may control the current running through the wireof actuator 770. Generally, an increased current will result in anincrease in magnetic field strength, thereby moving permanent magnet 726more quickly and increasing the haptic output's strength.

The processor may control additional aspects of electromagnet 770. Forexample, upon receipt of force sensor's 717 signal, the processor maypower up the electromagnet and/or alter the state of the electromagnetso as to ready electromagnet 770 to generate a magnetic field to actuatebutton 710. Such a scenario may occur if the electromagnet isconsistently powered on but at a level that generates a magnetic fieldof a size and/or strength that does not pivot permanent magnet 726. Uponreceipt of the output signal from force sensor 717, the processor maycontrol electromagnet 770 to move from stand-by status to a fullpower-on mode, thereby actuating button 710 by moving permanent magnet726. In some embodiments, input device 701 may be configured to actuate(e.g., the button moves) only upon receiving an input exceeding athreshold force level, below which no actuation is triggered.

In some embodiments, motion of permanent magnet 726 within electromagnet770 may be sensed by measuring the back electromotive force (EMF) of theelectromagnet.

Generally, the EMF induced in the electromagnet will vary with amagnitude of permanent magnet's 726 travel. Further, as a user pressesharder on input surface 712 or otherwise more rigidly constrains theinput surface with his or her finger, the permanent magnet's travelreduces. Thus, if a user has a “stiff input, input structure 710 (e.g.,button) travel is constrained and this may be sensed by measuring theback EMF of electromagnet 770 via a sensor. A user may provide a stiffinput if the user is exerting high force on input surface 712, iswearing gloves, has dry skin, a calloused finger, and so on. Generally,conditions that yield a stiff input also reduce sensitivity to hapticoutput. Accordingly, when the back EMF of the electromagnet 770 isexceeds a threshold, a processing unit of input device 701 may directadditional power to the electromagnet 770 to increase the force andperceptibility of haptic output.

Button 710 may be positioned in an opening along an exterior surface ofan electronic device, such that the button presents an input surface toa user. Button 710 may be conformal with the exterior surface, or mayproject from the exterior surface of a host electronic device. In oneembodiment, button 710 is oblong and fits along an exterior edge of anelectronic device, such as a mobile phone.

Button 710 may actuate (e.g., move) in any of several ways. In theembodiment of FIGS. 7A-7C, button 710 pivots off major axis 739 of theinput device 701, which is generally perpendicular to its pivot axis.However, other configurations are possible. For example, button 710 maybe configured to actuate along a minor axis. In some embodiments, button710 may actuate in a seesaw manner. In some embodiments, button 710moves along a surface or edge of a host electronic device.

In one embodiment, although button 710 may actuate, the actuation is notrequired to register a button input to an electronic device, such as toregister a button input by a processor of an electronic device. Statedanother way, the physical movement or actuation of button 710 is notrequired to register a button on or off input. Instead, button actuationis effected to provide a type of haptic feedback to the user.

Button 710 may have a variety of shapes, including defining a curved orconvex input surface 712, and/or may be rectangular, square, and so on.As another example, input surface 712 may be substantially flat. Inputsurface 712 and/or other parts of button 710 may include texture such asbumps, ridges, or the like. Button 710 may have readjusted, beveled, orflat edges. Generally, the smaller the curvature of input surface 712,the greater the shear (e.g., transverse displacement) of the user's skincontacting the input surface and thus the greater the perceptibility ofthe haptic output. Accordingly, travel of planar input surfaces 712 maybe more easily perceived by a user than the same travel of a curvedinput surface. The curvature of the input surface 712 may be selected toimpart a particular haptic output or particular perceptibility of ahaptic output.

Generally, if the curvature of input surface 712 equals the curvature ofan arc segment along which input surface 712 travels during rotation ofthe input structure 710 about the pivot, the skin of a user's finger incontact with the input surface 712 experiences purely tangential motionfrom the input surface. The “arc segment” is the portion of a circlethrough which a point on the input surface moves while the inputstructure rotates. Put another way, if every point of input surface 712lies on a single arc circumscribed by the entirety of input surface 712while haptic output is provided, then the curvature of the input surfaceequals the curvature of an arc segment. Put still another way, if thedistance from pivot 727 to every point of the input surface within therotational plane is equal, then the curvature of the input surface 712matches the curvature of the arc segment during rotation. Purelytangential motion of the input surface 712 against a user's skin yieldsa high degree of skin shear and a unique feeling of haptic output.Generally, such haptic output is indistinguishable ornear-indistinguishable from a “click” or depress of a typical buttonthat moves in the direction of an input force.

By changing the curvature of input surface 712, the feel of the hapticoutput may be varied. The more the curvature of the input surface variesfrom the arc segment along which the input surface 712 travels duringrotation, the more the haptic output feels like a “rocking” (e.g., forsimulating a “toggle” operations) motion to a user as opposed to a“clicking” or depressing/collapsing motion. The curvature of the inputsurface 712 may be tuned to provide particular haptic outputs, asdesired or necessary.

FIG. 3 is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments,device 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). Device 300 typically includesone or more processing units (CPU's) 310, one or more network or othercommunications interfaces 360, memory 370, and one or more communicationbuses 320 for interconnecting these components. Communication buses 320optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.Device 300 includes input/output (I/O) interface 330 comprising display340, which is typically a touch-screen display. I/O interface 330 alsooptionally includes a keyboard and/or mouse (or other pointing device)350 and touchpad 355, tactile output generator 357 for generatingtactile outputs on device 300 (e.g., similar to tactile outputgenerator(s) 167 described above with reference to FIG. 1A), sensors 359(e.g., optical, acceleration, proximity, touch-sensitive, and/or contactintensity sensors similar to contact intensity sensor(s) 165 describedabove with reference to FIG. 1A). Memory 370 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices; and optionally includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid state storagedevices. Memory 370 optionally includes one or more storage devicesremotely located from CPU(s) 310. In some embodiments, memory 370 storesprograms, modules, and data structures analogous to the programs,modules, and data structures stored in memory 102 of portablemultifunction device 100 (FIG. 1A), or a subset thereof. Furthermore,memory 370 optionally stores additional programs, modules, and datastructures not present in memory 102 of portable multifunction device100. For example, memory 370 of device 300 optionally stores drawingmodule 380, presentation module 382, word processing module 384, websitecreation module 386, disk authoring module 388, and/or spreadsheetmodule 390, while memory 102 of portable multifunction device 100 (FIG.1A) optionally does not store these modules.

Each of the above identified elements in FIG. 3 are, optionally, storedin one or more of the previously mentioned memory devices. Each of theabove identified modules corresponds to a set of instructions forperforming a function described above. The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules are, optionally, combined or otherwisere-arranged in various embodiments. In some embodiments, memory 370optionally stores a subset of the modules and data structures identifiedabove. Furthermore, memory 370 optionally stores additional modules anddata structures not described above.

Attention is now directed towards embodiments of user interfaces (“UI”)that are, optionally, implemented on portable multifunction device 100.

FIG. 4A illustrates a portable multifunction device 100 with one or moreintensity-sensitive off-display input regions on one or more peripheralsides of the device, and a touch-screen display showing an example homescreen user interface with a menu of application launch iconscorresponding to different applications, in accordance with someembodiments. Similar user interfaces are, optionally, implemented ondevice 300. In some embodiments, user interface 400 includes thefollowing elements, or a subset or superset thereof:

-   -   Signal strength indicator(s) 402 for wireless communication(s),        such as cellular and Wi-Fi signals;    -   Time 404;    -   a Bluetooth indicator;    -   Battery status indicator 406;    -   Tray 408 with icons for frequently used applications, such as:        -   Icon 416 for telephone module 138, labeled “Phone,” which            optionally includes an indicator 414 of the number of missed            calls or voicemail messages;        -   Icon 418 for e-mail client module 140, labeled “Mail,” which            optionally includes an indicator 410 of the number of unread            e-mails;        -   Icon 420 for browser module 147, labeled “Browser;” and        -   Icon 422 for video and music player module 152, also            referred to as iPod (trademark of Apple Inc.) module 152,            labeled “iPod;” and    -   Icons for other applications, such as:        -   Icon 424 for IM module 141, labeled “Messages;”        -   Icon 426 for calendar module 148, labeled “Calendar;”        -   Icon 428 for image management module 144, labeled “Photos;”        -   Icon 430 for camera module 143, labeled “Camera;”        -   Icon 432 for online video module 155, labeled “Online            Video;”        -   Icon 434 for stocks widget 149-2, labeled “Stocks;”        -   Icon 436 for map module 154, labeled “Maps;”        -   Icon 438 for weather widget 149-1, labeled “Weather;”        -   Icon 440 for alarm clock widget 149-4, labeled “Clock;”        -   Icon 442 for workout support module 142, labeled “Workout            Support;”        -   Icon 444 for notes module 153, labeled “Notes;” and        -   Icon 446 for a settings application or module, which            provides access to settings for device 100 and its various            applications 136.

It should be noted that the icon labels illustrated in FIG. 4A aremerely examples. For example, in some embodiments, icon 422 for videoand music player module 152 is labeled “Music” or “Music Player.” Otherlabels are, optionally, used for various application icons. In someembodiments, a label for a respective application icon includes a nameof an application corresponding to the respective application icon. Insome embodiments, a label for a particular application icon is distinctfrom a name of an application corresponding to the particularapplication icon. In some embodiments, when an icon corresponding to arespective application is activated (e.g., by a tap input), the devicelaunches the respective application and replaces the home screen userinterface with a user interface of the respective application. When arequest for dismissing the currently displayed user interface andreturning to the home screen user interface is received (e.g., apredefined home gesture or activation of a home button is detected), thedevice dismisses the currently displayed user interface and redisplaysthe home screen user interface (e.g., a last displayed page of the homescreen user interface or a default starting page of the home screen userinterface).

As shown in FIG. 4A, in some embodiments, the portable multifunctiondevice 100 does not include a home button on the display-side of thedevice, and the touch-screen 112 occupies substantially all of availableareas on the display-side of the device. The device 100 as shown in FIG.4A includes a number of discrete intensity-sensitive input regions 455,456, and 457. The input regions 456 and 457 are optionally implementedon two sub-portions of a contiguous touch-sensitive surface. Each of theinput regions are coupled with a respective set of one or more localizedtactile output generators for generating localized tactile outputs thathave greater amplitudes at the input region (or a respective sub-portionthereof), and smaller amplitudes or negligible amplitudes at locationsoutside of the input region (or the respective sub-portion thereof).

FIG. 4B illustrates an example user interface on a device (e.g., device300, FIG. 3) with a touch-sensitive surface 451 (e.g., a tablet ortouchpad 355, FIG. 3) that is separate from the display 450. Device 300also, optionally, includes one or more contact intensity sensors (e.g.,one or more of sensors 357) for detecting intensities of contacts ontouch-sensitive surface 451 and/or one or more tactile output generators359 for generating tactile outputs for a user of device 300. In someembodiments, the device includes an intensity-sensitive input regionthat is distinct from touch-sensitive surface 451. For example, theintensity-sensitive input region is disposed on a peripheral side of thedevice (e.g., a side edge of the device) that is adjacent to thedisplay-side of the device, and the touch-sensitive surface 451 iscoplanar with the display-side of the device or disposed on thedisplay-side of the device. In some embodiments, a contact detected onthe intensity-sensitive input regions do not have a corresponding focusselector displayed on the display of the device that is located at acorresponding location of the contact on the input region and/or thatmove in accordance with movement of the contact across the input region.In some embodiments, the touch-sensitive surface 451 serves as anoff-display input region of the device.

Although many of the examples that follow will be given with referenceto inputs on touch screen display 112 (where the touch sensitive surfaceand the display are combined), in some embodiments, the device detectsinputs on a touch-sensitive surface that is separate from the display,as shown in FIG. 4B. In some embodiments, the touch-sensitive surface(e.g., 451 in FIG. 4B) has a primary axis (e.g., 452 in FIG. 4B) thatcorresponds to a primary axis (e.g., 453 in FIG. 4B) on the display(e.g., 450). In accordance with these embodiments, the device detectscontacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface451 at locations that correspond to respective locations on the display(e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470).In this way, user inputs (e.g., contacts 460 and 462, and movementsthereof) detected by the device on the touch-sensitive surface (e.g.,451 in FIG. 4B) are used by the device to manipulate the user interfaceon the display (e.g., 450 in FIG. 4B) of the multifunction device whenthe touch-sensitive surface is separate from the display. It should beunderstood that similar methods are, optionally, used for other userinterfaces described herein.

Additionally, while the following examples are given primarily withreference to finger inputs (e.g., finger contacts, finger tap gestures,finger swipe gestures, etc.), it should be understood that, in someembodiments, one or more of the finger inputs are replaced with inputfrom another input device (e.g., a mouse based input or a stylus input).For example, a swipe gesture is, optionally, replaced with a mouse click(e.g., instead of a contact) followed by movement of the cursor alongthe path of the swipe (e.g., instead of movement of the contact). Asanother example, a tap gesture is, optionally, replaced with a mouseclick while the cursor is located over the location of the tap gesture(e.g., instead of detection of the contact followed by ceasing to detectthe contact). Similarly, when multiple user inputs are simultaneouslydetected, it should be understood that multiple computer mice are,optionally, used simultaneously, or a mouse and finger contacts are,optionally, used simultaneously.

As used herein, the term “focus selector” is an input element thatindicates a current part of a user interface with which a user isinteracting. In some implementations that include a cursor or otherlocation marker, the cursor acts as a “focus selector,” so that when aninput (e.g., a press input) is detected on a touch-sensitive surface(e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B)while the cursor is over a particular user interface element (e.g., abutton, window, slider or other user interface element), the particularuser interface element is adjusted in accordance with the detectedinput. In some implementations that include a touch-screen display(e.g., touch-sensitive display system 112 in FIG. 1A or the touch screenin FIG. 4A) that enables direct interaction with user interface elementson the touch-screen display, a detected contact on the touch-screen actsas a “focus selector,” so that when an input (e.g., a press input by thecontact) is detected on the touch-screen display at a location of aparticular user interface element (e.g., a button, window, slider orother user interface element), the particular user interface element isadjusted in accordance with the detected input. In some implementations,focus is moved from one region of a user interface to another region ofthe user interface without corresponding movement of a cursor ormovement of a contact on a touch-screen display (e.g., by using a tabkey or arrow keys to move focus from one button to another button); inthese implementations, the focus selector moves in accordance withmovement of focus between different regions of the user interface.Without regard to the specific form taken by the focus selector, thefocus selector is generally the user interface element (or contact on atouch-screen display) that is controlled by the user so as tocommunicate the user's intended interaction with the user interface(e.g., by indicating, to the device, the element of the user interfacewith which the user is intending to interact). For example, the locationof a focus selector (e.g., a cursor, a contact, or a selection box) overa respective button while a press input is detected on thetouch-sensitive surface (e.g., a touchpad or touch screen) will indicatethat the user is intending to activate the respective button (as opposedto other user interface elements shown on a display of the device).

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface is the force or pressure (force perunit area) of a contact (e.g., a finger contact or a stylus contact) onthe touch-sensitive surface, or to a substitute (proxy) for the force orpressure of a contact on the touch-sensitive surface. The intensity of acontact has a range of values that includes at least four distinctvalues and more typically includes hundreds of distinct values (e.g., atleast 256). Intensity of a contact is, optionally, determined (ormeasured) using various approaches and various sensors or combinationsof sensors. For example, one or more force sensors underneath oradjacent to the touch-sensitive surface are, optionally, used to measureforce at various points on the touch-sensitive surface. In someimplementations, force measurements from multiple force sensors arecombined (e.g., a weighted average or a sum) to determine an estimatedforce of a contact. Similarly, a pressure-sensitive tip of a stylus is,optionally, used to determine a pressure of the stylus on thetouch-sensitive surface. Alternatively, the size of the contact areadetected on the touch-sensitive surface and/or changes thereto, thecapacitance of the touch-sensitive surface proximate to the contactand/or changes thereto, and/or the resistance of the touch-sensitivesurface proximate to the contact and/or changes thereto are, optionally,used as a substitute for the force or pressure of the contact on thetouch-sensitive surface. In some implementations, the substitutemeasurements for contact force or pressure are used directly todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is described in units corresponding to thesubstitute measurements). In some implementations, the substitutemeasurements for contact force or pressure are converted to an estimatedforce or pressure and the estimated force or pressure is used todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is a pressure threshold measured in units ofpressure). Using the intensity of a contact as an attribute of a userinput allows for user access to additional device functionality that mayotherwise not be readily accessible by the user on a reduced-size devicewith limited real estate for displaying affordances (e.g., on atouch-sensitive display) and/or receiving user input (e.g., via atouch-sensitive display, a touch-sensitive surface, or aphysical/mechanical control such as a knob or a button).

In some embodiments, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some embodiments, at least a subset of the intensitythresholds are determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of device 100). For example, a mouse “click”threshold of a trackpad or touch-screen display can be set to any of alarge range of predefined thresholds values without changing thetrackpad or touch-screen display hardware. Additionally, in someimplementations a user of the device is provided with software settingsfor adjusting one or more of the set of intensity thresholds (e.g., byadjusting individual intensity thresholds and/or by adjusting aplurality of intensity thresholds at once with a system-level click“intensity” parameter).

As used in the specification and claims, the term “characteristicintensity” of a contact is a characteristic of the contact based on oneor more intensities of the contact. In some embodiments, thecharacteristic intensity is based on multiple intensity samples. Thecharacteristic intensity is, optionally, based on a predefined number ofintensity samples, or a set of intensity samples collected during apredetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10seconds) relative to a predefined event (e.g., after detecting thecontact, prior to detecting liftoff of the contact, before or afterdetecting a start of movement of the contact, prior to detecting an endof the contact, before or after detecting an increase in intensity ofthe contact, and/or before or after detecting a decrease in intensity ofthe contact). A characteristic intensity of a contact is, optionallybased on one or more of: a maximum value of the intensities of thecontact, a mean value of the intensities of the contact, an averagevalue of the intensities of the contact, a top 10 percentile value ofthe intensities of the contact, a value at the half maximum of theintensities of the contact, a value at the 90 percent maximum of theintensities of the contact, a value produced by low-pass filtering theintensity of the contact over a predefined period or starting at apredefined time, or the like. In some embodiments, the duration of thecontact is used in determining the characteristic intensity (e.g., whenthe characteristic intensity is an average of the intensity of thecontact over time). In some embodiments, the characteristic intensity iscompared to a set of one or more intensity thresholds to determinewhether an operation has been performed by a user. For example, the setof one or more intensity thresholds may include a first intensitythreshold and a second intensity threshold. In this example, a contactwith a characteristic intensity that does not exceed the first thresholdresults in a first operation, a contact with a characteristic intensitythat exceeds the first intensity threshold and does not exceed thesecond intensity threshold results in a second operation, and a contactwith a characteristic intensity that exceeds the second intensitythreshold results in a third operation. In some embodiments, acomparison between the characteristic intensity and one or moreintensity thresholds is used to determine whether or not to perform oneor more operations (e.g., whether to perform a respective option orforgo performing the respective operation) rather than being used todetermine whether to perform a first operation or a second operation.

In some embodiments, a portion of a gesture is identified for purposesof determining a characteristic intensity. For example, atouch-sensitive surface may receive a continuous swipe contacttransitioning from a start location and reaching an end location (e.g.,a drag gesture), at which point the intensity of the contact increases.In this example, the characteristic intensity of the contact at the endlocation may be based on only a portion of the continuous swipe contact,and not the entire swipe contact (e.g., only the portion of the swipecontact at the end location). In some embodiments, a smoothing algorithmmay be applied to the intensities of the swipe contact prior todetermining the characteristic intensity of the contact. For example,the smoothing algorithm optionally includes one or more of: anunweighted sliding-average smoothing algorithm, a triangular smoothingalgorithm, a median filter smoothing algorithm, and/or an exponentialsmoothing algorithm. In some circumstances, these smoothing algorithmseliminate narrow spikes or dips in the intensities of the swipe contactfor purposes of determining a characteristic intensity.

The user interface figures described herein optionally include variousintensity diagrams that show the current intensity of the contact on thetouch-sensitive surface relative to one or more intensity thresholds(e.g., a contact detection intensity threshold IT₀, a hint intensitythreshold IT_(H) (e.g., that is higher than IT₀), a light pressintensity threshold IT_(L) (e.g., that is at least initially higher thanIT_(H)), a deep press intensity threshold IT_(D) (e.g., that is at leastinitially higher than IT_(L)), and/or one or more other intensitythresholds (e.g., an intensity threshold IT_(H) that is lower thanIT_(L))). This intensity diagram is typically not part of the displayeduser interface, but is provided to aid in the interpretation of thefigures. In some embodiments, the light press intensity thresholdcorresponds to an intensity at which the device will perform operationstypically associated with clicking a button of a physical mouse or atrackpad. In some embodiments, the deep press intensity thresholdcorresponds to an intensity at which the device will perform operationsthat are different from operations typically associated with clicking abutton of a physical mouse or a trackpad. In some embodiments, when acontact is detected with a characteristic intensity below the lightpress intensity threshold (e.g., and above a nominal contact-detectionintensity threshold IT₀ below which the contact is no longer detected),the device will move a focus selector in accordance with movement of thecontact on the touch-sensitive surface without performing an operationassociated with the light press intensity threshold or the deep pressintensity threshold. Generally, unless otherwise stated, these intensitythresholds are consistent between different sets of user interfacefigures.

In some embodiments, the response of the device to inputs detected bythe device depends on criteria based on the contact intensity during theinput. For example, for some “light press” inputs, the intensity of acontact exceeding a first intensity threshold during the input triggersa first response. In some embodiments, the response of the device toinputs detected by the device depends on criteria that include both thecontact intensity during the input and time-based criteria. For example,for some “deep press” inputs, the intensity of a contact exceeding asecond intensity threshold during the input, greater than the firstintensity threshold for a light press, triggers a second response onlyif a delay time has elapsed between meeting the first intensitythreshold and meeting the second intensity threshold. This delay time istypically less than 200 ms in duration (e.g., 40, 100, or 120 ms,depending on the magnitude of the second intensity threshold, with thedelay time increasing as the second intensity threshold increases). Thisdelay time helps to avoid accidental recognition of deep press inputs.As another example, for some “deep press” inputs, there is areduced-sensitivity time period that occurs after the time at which thefirst intensity threshold is met. During the reduced-sensitivity timeperiod, the second intensity threshold is increased. This temporaryincrease in the second intensity threshold also helps to avoidaccidental deep press inputs. For other deep press inputs, the responseto detection of a deep press input does not depend on time-basedcriteria.

In some embodiments, one or more of the input intensity thresholdsand/or the corresponding outputs vary based on one or more factors, suchas user settings, contact motion, input timing, application running,rate at which the intensity is applied, number of concurrent inputs,user history, environmental factors (e.g., ambient noise), focusselector position, and the like. Example factors are described in U.S.patent application Ser. Nos. 14/399,606 and 14/624,296, which areincorporated by reference herein in their entireties.

For example, FIG. 4C illustrates a dynamic intensity threshold 480 thatchanges over time based in part on the intensity of touch input 476 overtime. Dynamic intensity threshold 480 is a sum of two components, firstcomponent 474 that decays over time after a predefined delay time p1from when touch input 476 is initially detected, and second component478 that trails the intensity of touch input 476 over time. The initialhigh intensity threshold of first component 474 reduces accidentaltriggering of a “deep press” response, while still allowing an immediate“deep press” response if touch input 476 provides sufficient intensity.Second component 478 reduces unintentional triggering of a “deep press”response by gradual intensity fluctuations of in a touch input. In someembodiments, when touch input 476 satisfies dynamic intensity threshold480 (e.g., at point 481 in FIG. 4C), the “deep press” response istriggered.

FIG. 4D illustrates another dynamic intensity threshold 486 (e.g.,intensity threshold I_(D) (e.g., also referred to as IT_(D))). FIG. 4Dalso illustrates two other intensity thresholds: a first intensitythreshold I_(H) (e.g., also referred to as IT_(H)) and a secondintensity threshold I_(L). In FIG. 4D, although touch input 484satisfies the first intensity threshold I_(H) and the second intensitythreshold I_(L) (e.g., also referred to as IT_(L)) prior to time p2, noresponse is provided until delay time p2 has elapsed at time 482. Alsoin FIG. 4D, dynamic intensity threshold 486 decays over time, with thedecay starting at time 488 after a predefined delay time p1 has elapsedfrom time 482 (when the response associated with the second intensitythreshold I_(L) was triggered). This type of dynamic intensity thresholdreduces accidental triggering of a response associated with the dynamicintensity threshold I_(D) immediately after, or concurrently with,triggering a response associated with a lower intensity threshold, suchas the first intensity threshold I_(H) or the second intensity thresholdI_(L).

FIG. 4E illustrate yet another dynamic intensity threshold 492 (e.g.,intensity threshold I_(D)). In FIG. 4E, a response associated with theintensity threshold I_(L) is triggered after the delay time p2 haselapsed from when touch input 490 is initially detected. Concurrently,dynamic intensity threshold 492 decays after the predefined delay timep1 has elapsed from when touch input 490 is initially detected. So adecrease in intensity of touch input 490 after triggering the responseassociated with the intensity threshold I_(L), followed by an increasein the intensity of touch input 490, without releasing touch input 490,can trigger a response associated with the intensity threshold I_(D)(e.g., at time 494) even when the intensity of touch input 490 is belowanother intensity threshold, for example, the intensity threshold I_(L).

An increase of characteristic intensity of the contact from an intensitybelow the light press intensity threshold IT_(L) to an intensity betweenthe light press intensity threshold IT_(L) and the deep press intensitythreshold IT_(D) is sometimes referred to as a “light press” input. Anincrease of characteristic intensity of the contact from an intensitybelow the deep press intensity threshold IT_(D) to an intensity abovethe deep press intensity threshold IT_(D) is sometimes referred to as a“deep press” input. An increase of characteristic intensity of thecontact from an intensity below the contact-detection intensitythreshold IT₀ to an intensity between the contact-detection intensitythreshold IT₀ and the light press intensity threshold IT_(L) issometimes referred to as detecting the contact on the touch-surface. Adecrease of characteristic intensity of the contact from an intensityabove the contact-detection intensity threshold IT₀ to an intensitybelow the contact-detection intensity threshold IT₀ is sometimesreferred to as detecting liftoff of the contact from the touch-surface.In some embodiments, IT₀ is zero. In some embodiments, IT₀ is greaterthan zero. In some illustrations, a shaded circle or oval is used torepresent intensity of a contact on the touch-sensitive surface. In someillustrations, a circle or oval without shading is used represent arespective contact on the touch-sensitive surface without specifying theintensity of the respective contact.

In some embodiments, described herein, one or more operations areperformed in response to detecting a gesture that includes a respectivepress input or in response to detecting the respective press inputperformed with a respective contact (or a plurality of contacts), wherethe respective press input is detected based at least in part ondetecting an increase in intensity of the contact (or plurality ofcontacts) above a press-input intensity threshold. In some embodiments,the respective operation is performed in response to detecting theincrease in intensity of the respective contact above the press-inputintensity threshold (e.g., the respective operation is performed on a“down stroke” of the respective press input). In some embodiments, thepress input includes an increase in intensity of the respective contactabove the press-input intensity threshold and a subsequent decrease inintensity of the contact below the press-input intensity threshold, andthe respective operation is performed in response to detecting thesubsequent decrease in intensity of the respective contact below thepress-input threshold (e.g., the respective operation is performed on an“up stroke” of the respective press input).

In some embodiments, the device employs intensity hysteresis to avoidaccidental inputs sometimes termed “jitter,” where the device defines orselects a hysteresis intensity threshold with a predefined relationshipto the press-input intensity threshold (e.g., the hysteresis intensitythreshold is X intensity units lower than the press-input intensitythreshold or the hysteresis intensity threshold is 75%, 90%, or somereasonable proportion of the press-input intensity threshold). Thus, insome embodiments, the press input includes an increase in intensity ofthe respective contact above the press-input intensity threshold and asubsequent decrease in intensity of the contact below the hysteresisintensity threshold that corresponds to the press-input intensitythreshold, and the respective operation is performed in response todetecting the subsequent decrease in intensity of the respective contactbelow the hysteresis intensity threshold (e.g., the respective operationis performed on an “up stroke” of the respective press input).Similarly, in some embodiments, the press input is detected only whenthe device detects an increase in intensity of the contact from anintensity at or below the hysteresis intensity threshold to an intensityat or above the press-input intensity threshold and, optionally, asubsequent decrease in intensity of the contact to an intensity at orbelow the hysteresis intensity, and the respective operation isperformed in response to detecting the press input (e.g., the increasein intensity of the contact or the decrease in intensity of the contact,depending on the circumstances).

For ease of explanation, the description of operations performed inresponse to a press input associated with a press-input intensitythreshold or in response to a gesture including the press input are,optionally, triggered in response to detecting: an increase in intensityof a contact above the press-input intensity threshold, an increase inintensity of a contact from an intensity below the hysteresis intensitythreshold to an intensity above the press-input intensity threshold, adecrease in intensity of the contact below the press-input intensitythreshold, or a decrease in intensity of the contact below thehysteresis intensity threshold corresponding to the press-inputintensity threshold. Additionally, in examples where an operation isdescribed as being performed in response to detecting a decrease inintensity of a contact below the press-input intensity threshold, theoperation is, optionally, performed in response to detecting a decreasein intensity of the contact below a hysteresis intensity thresholdcorresponding to, and lower than, the press-input intensity threshold.As described above, in some embodiments, the triggering of theseresponses also depends on time-based criteria being met (e.g., a delaytime has elapsed between a first intensity threshold being met and asecond intensity threshold being met).

User Interfaces and Associated Processes

Attention is now directed towards embodiments of user interfaces (“UI”)and associated processes that may be implemented on an electronicdevice, such as portable multifunction device 100 or device 300, with adisplay, a touch-sensitive surface, one or more tactile outputgenerators for generating tactile outputs (e.g., one or more localizedtactile output generators and/or one or more whole device tactile outputgenerators), one or more sensors to detect intensities of contacts withthe touch-sensitive surface and one or more input regions of the device(e.g., one or more off-display side buttons).

FIGS. 5A1-5H15 illustrate exemplary user interfaces for providingtactile outputs and visual feedback in response to multiple types ofinputs on an intensity-sensitive input region (e.g., an off-displayinput regions 455, 456, and 457) of the device in accordance with someembodiments. The user interfaces in these figures are used to illustratethe processes described below, including the processes in FIGS. 6A-6D,7A-7E, 8A-8D, 9A-9C, 10A-10C, 11A-11B, and 12A-12E. For convenience ofexplanation, some of the embodiments will be discussed with reference tooperations performed on a device with a touch-sensitive display system112. In such embodiments, the focus selector is, optionally: arespective finger or stylus contact, a representative pointcorresponding to a finger or stylus contact (e.g., a centroid of arespective contact or a point associated with a respective contact), ora centroid of two or more contacts detected on the touch-sensitivedisplay system 112. However, analogous operations are, optionally,performed on a device with a display 450 and a separate touch-sensitivesurface 451 in response to detecting the contacts on the touch-sensitivesurface 451 while displaying the user interfaces shown in the figures onthe display 450, along with a focus selector.

FIGS. 5A1-5A8 illustrate exemplary user interfaces for providing hapticand visual feedback for button interactions corresponding to queryingand/or modifying a current state of a device attribute in accordancewith some embodiments. The user interfaces in these figures are used toillustrate the processes described below, including the processes inFIGS. 6A-6D and 12A-12E.

FIG. 5A1 illustrates device 100 in a display-off state. For example,after a period of inactivity, the device automatically turns off thedisplay 112 to conserve power. In some embodiments, the device is lockedwhen the display is turned off and requires authentication (e.g., viafacial recognition or biometric inputs) to be unlocked. No userinterface is displayed on the screen when the device is in thedisplay-off state. In some embodiments, the device displays a wakescreen user interface (e.g., wake screen user interface 501) as aninitial interface that is displayed when the device transitions from thedisplay-off state to a display-on state (e.g., in response to occurrenceof an event or detection of a user input).

FIG. 5A2 illustrates that, when a new notification (e.g., notification502 for a newly received message) is generated at device 100, the devicetransitions from the display-off state to the display-on state, anddisplays the new notification 502 on wake screen user interface 501. Thewake screen user interface displays an existing unread notification 504that was generated at an earlier time, but has not been read orinteracted with by the user. The wake screen user interface 501 displaysthe current time. The wake screen user interface 501 is dismissed whenan upward wipe gesture (e.g., a home-display gesture) is detected at thebottom edge of the touch-screen 112, and a home screen user interface ora user interface of a last displayed application replaces the wakescreen user interface 501 on the screen. In some embodiments, the devicenavigates to a mini application user interface in response to ahorizontal swipe gesture detected on the touch-screen, withoutdisplaying the home screen or the last displayed application userinterface.

FIG. 5A3 illustrates that, after a period of inactivity by the usersince the receipt of new notification 502, the device has returned tothe display-off state, without changing the read/unread statuses ofnotifications 502 and 504. In FIG. 5A3, a touch input by contact 506 isdetected on input region 456. In accordance with the detection of thetouch input by contact 506 (e.g., intensity of contact 506 exceedscontact detection intensity threshold IT₀), without requiring theintensity of contact 506 to exceed a light press intensity thresholdIT_(L), the device generates localized tactile outputs 508 at inputregion 456 to indicate a current value or state (e.g., “Present, asopposed to “Not Present”) of a predefined device attribute (e.g., thepresence/absence of unread notifications at the device). In someembodiments, tactile outputs 508 further indicate the number of unreadnotifications (e.g., two) present at the device based on one or morecharacteristics of the tactile outputs (e.g., the tactile outputs 508are a tactile output sequence with a corresponding number of identical,discrete tactile outputs at input region 456). In some embodiments, thedevice generates tactile outputs 508 only after contact 506 has beenmaintained on input region 456 for at least a threshold amount of timeT, to reduce accidental triggering of the device's response regardingthe presence/absence of unread notifications. In some embodiments, thedevice does not transition from the display-off state to the display-onstate when the intensity of the input by contact 506 remains below thehint intensity threshold IT_(H) and the light press intensity thresholdIT_(L). In some embodiments, the device does not transition from thedisplay-off state to the display-on state when the device detects thatthe device is inside of a pocket (e.g., based on inputs from an ambientlight sensor). Forgoing turning on the display when responding to theuser's query of device attribute information conserves energy and bettermaintains user privacy (e.g., the device provides the information thatthe user needs when the device remains concealed inside the user'spocket).

FIG. 5A4 illustrates that, in some embodiments, wake screen userinterface 501 gradually fades in (e.g., in a shaded state) in when anintensity of the input by contact 506 on input region 456 increasesabove the hint intensity threshold IT_(H). The visibility of wake screenuser interface along with the unread notifications 502 and 504dynamically changes in accordance with a current intensity of contact506 on input region 456. In some embodiments, tactile outputs 508 isprovided in conjunction with the visual feedback (e.g., visual displayof the unread notifications 502 and 504 and/or a count thereof)displayed on the screen. In some embodiments, no tactile outputindicating the presence/count of unread notifications is provided untilthe intensity of contact 506 exceeds the hint intensity thresholdIT_(H). In some embodiments, the device forgoes requiring that thecontact be maintained on input region 456 for at least a thresholdamount of time T and provides the device attribute regarding unreadnotifications to the user if the intensity of the contact exceeds thehint intensity threshold IT_(H) before the time threshold T is met.

In FIGS. 5A3 and 5A4, the device generates visual feedback and hapticfeedback regarding the current value or state of a device attribute,namely, the presence of unread notifications, of the device, withoutchanging the current value or state of the device attribute, because theintensity of the input by contact 506 has not exceeded a first intensitythreshold (e.g., the light press intensity threshold IT_(L)). In someembodiments, when termination of the input by contact 506 (e.g.,lift-off of contact 506 from input region 456) is detected without theintensity of contact 506 having exceeded the first intensity threshold,the device transitions back to the display-off state (e.g., followingthe state shown in FIG. 5A4) and notifications 502 and 504 maintainstheir unread statuses.

FIG. 5A5 illustrates that, before termination of the input by contact506 is detected, intensity of contact 506 on input region 456 increasesabove the first intensity threshold (e.g., the light press intensitythreshold IT_(L)). In response to detecting that the intensity ofcontact 506 has increased above the first intensity threshold, device100 completes its transition to the display-on state, and displaysnotifications 502 and 504 on wake screen user interface 501. The devicealso changes the read/unread statuses of notifications 502 and 504 from“unread” to “read.” In some embodiments, the device generates localizedtactile output 510 (e.g., with a tactile output profile that isdifferent from the tactile output profile for tactile outputs 508) toindicate the changes made to the read/unread statuses of notifications502 and 504, and the change in the current state of the deviceparameter, namely, the presence/absence of unread notifications, from“Present” to “Not Present”.

In some embodiments, if the unread notifications displayed on wakescreen user interface 501 has corresponding long versions, whenintensity of input by contact 506 increases above a second intensitythreshold (e.g., a deep press intensity threshold IT_(D)), the devicereplaces notifications 502 and 504 with corresponding long versions 502′and 504′ of the notifications, as shown in FIG. 5A6. The long versionsof the notifications include additional content (e.g., an attachedimage, or a voicemail recording) or functionality (e.g., a playbackbutton) that were not included in notifications 502 and 504. In someembodiments, the device optionally generates another localized tactileoutput 512 (e.g., with a tactile output profile that is distinct fromthe respective tactile output profiles of tactile outputs 508 and 510)at input region 456 in response to the intensity of the input by contact512 exceeding the second intensity threshold.

FIG. 5A7 illustrates that, after a period of inactivity by the user, thedevice returns to the display-off state. The notifications 502 and 504are cleared from the wake screen user interface (e.g., saved tonotification history or cleared) because they are no longer “unread”notifications as a result of the interactions shown in FIGS. 5A5 and5A6. FIG. 5A8 illustrates, that when another input by contact 514 isdetected on input region 456, and the device-query criteria forproviding the current value or state of the predefined device attribute(e.g., the presence/absence of unread notifications) of the device aremet by the input by contact 514 (e.g., intensity of contact 514 exceededthe hint intensity threshold IT_(H)), the device responds by displayingthe wake screen user interface without any unread notifications (e.g.,the wake screen user interface 501 fades in by an amount in accordancewith the intensity of contact 514), and by forgoing generation of anylocalized tactile outputs at input region 456, to indicate that there iscurrently no unread notifications present at the device. In someembodiments, the device provides a predefined failure tactile output(e.g., a weak tactile output with a distinct tactile output profile(e.g., a failure buzz)) to indicate that the current value of thepredefined device attribute is “Not Present.” When the current value ofthe predefined device attribute is “Not Present,” the device does notchange the value of the device attribute to “Present” in response to anincrease in intensity of the input by contact 514 above the firstintensity threshold (e.g., the light press intensity threshold). In someembodiments, for a different device attribute with a toggle state (e.g.,the ringer ON/OFF state) or a continuously modifiable state (e.g., thevolume level), the device toggles the current value or state of thedevice attribute (e.g., from ON to OFF, or from OFF to ON) or continuesto change the current value of the device attribute (e.g., to incrementthe volume level by a respective amount) in response to an increase inintensity of the input by contact 514 above the first intensitythreshold (e.g., the light press intensity threshold).

FIGS. 5B1-5B11 illustrate exemplary user interfaces for providing hapticand visual feedback for button interactions corresponding to modifying adevice parameter in different manners in accordance with someembodiments. The user interfaces in these figures are used to illustratethe processes described below, including the processes in FIGS. 6A-6D8A-8D and 12A-12E.

In FIG. 5B1, user interface 516 of a browser application is displayed ontouch-screen 112. In FIG. 5B2, a touch input by contact 518 is detectedon input region 456 (e.g., in this example, input regions 456 and 457primarily serve as the volume up and volume down buttons of the device,respectively). In response to detecting the input by contact 518 oninput region 456 (e.g., intensity of contact 518 exceeds the detectionintensity threshold IT₀), and without requiring the intensity of theinput by contact 518 to exceed a first intensity threshold (e.g., thelight press intensity threshold IT_(L)), the device displays volumecontrol 520 (e.g., a slider control) with indicator 522 (e.g., a sliderindicator) indicating a current value of the device's volume setting. Insome embodiments, the background user interface 516 is darkened andblurred underneath volume control 520, and the amount of darkening andblurring is changed (e.g., increased or decreased) dynamically inaccordance with the current intensity of contact 518. In someembodiments, as the intensity of contact 518 remains below the firstintensity threshold (e.g., the light press intensity threshold IT_(L)),the current value of the volume setting does not change. In other words,the light touch on input region 456 causes the device to provide thecurrent value or state of a predefined device attribute (e.g., thedevice volume level), without changing the current value or state of thepredefined device attribute. In some embodiments, the device displaysvisual feedback (e.g., an animated arrow pointing upward) that indicatesthe direction in which the device attribute will be changed if a pressinput is detected on the input region 456 that is currently touched bythe user. In some embodiments, the device determines which deviceparameter (e.g., presence/absence of unread notification vs. devicevolume setting) is being queried by a particular light touch input oninput region 456 in accordance with context information (e.g., if thedevice is in the display-off state, the device responds withnotification status, and if the device is in the display-on state, thedevice responds with the current volume), without requiring presence ofa focus selector on the display.

FIG. 5B3 illustrates that, when the intensity of input by contact 518has increased above the first intensity threshold (e.g., the light pressintensity threshold IT_(L)), the device increases the current volume ofthe device (e.g., because the input region 456 corresponds to the volumeup function of the device) by a respective amount. In some embodiments,each time that the intensity of input by contact 518 increases above thefirst intensity threshold (e.g., a dynamic intensity threshold that isdetermined in accordance with the methods shown in FIGS. 4C-4E), thedevice increase the volume level of the device by a respective amount.In some embodiments, the exact amount of each increase is fixed. In someembodiments, the exactly amount of each increase is dynamicallydetermined, for example, in accordance of one or more characteristics(e.g., the speed and the total amount) of the increase in intensity whenthe first intensity threshold is crossed. In some embodiments, a fasterincrease in intensity across the first intensity threshold or a longerduration of an intensity maintained above the first intensity thresholdcauses a greater amount of increase in the volume level of the device.In some embodiments, for each increment made to the volume level inresponse to the increase in intensity of contact 518 above the firstintensity threshold, the device generates a respective tactile output(e.g., localized tactile output 524 (e.g., with a tactile output profilethat is different from the tactile output profiles of tactile outputs508, 510, and 512)). In some embodiments, the tactile output profile oftactile output 524 is optionally the same as the tactile output profileof tactile output 510, and corresponds to a light press input detectedon input region 456. In some embodiments, as shown in FIG. 5B3, whilethe volume level is changed by the input by contact 524, indicator 522moves up along volume control 520 to indicate the current volume levelof the device. User interface 516 remains darkened and blurredunderneath volume control 520 while contact 518 is continuously detectedon input region 456. In some embodiments, volume control 520 remainsdisplayed as long as contact 518 is detected on input region 456. Insome embodiments, when lift-off of contact 518 is detected after volumelevel of the device has been changed, the device cease to display volumecontrol 520 and appearance of user interface 516 is restored. The volumelevel of the device remains at the changed level after the volumecontrol is no longer displayed.

FIG. 5B4 illustrates that, when intensity of contact 518 increases abovethe first intensity threshold for a number of times, or the intensity ofcontact 518 has been maintained above the first intensity threshold foran extended period of time, the device has increased the volume level ofthe device to a maximum value of the device volume. In response to thevolume level of the device reaching the maximum value, the devicegenerates localized tactile output 526 at input region 456 (e.g., thetactile output is synchronized with the increase of intensity above thefirst intensity threshold or the satisfaction of the time threshold formaintaining the intensity above the first intensity threshold) toindicate that a maximum value of the volume setting has been reached. Insome embodiments, in addition to or in lieu of the localized tactileoutput 526 generated at input region 456, the device provides a wholedevice tactile output 528 to indicate that the maximum value of thevolume level has been reached.

FIGS. 5B5 and 5B6 illustrate that, in some embodiments, if an additionalpress input by the same contact or a different contact (e.g., contact530) is detected on input region 456, and the intensity of the contact(e.g., contact 530) exceeds the first intensity threshold, the devicedisplays (e.g., redisplays) volume control 520 showing the currentvolume level at the maximum value, but forgoes generating at least one(e.g., one or both) of localized tactile output 526 and whole tactileoutput 528, to indicate that no further change has been made to thevolume level in response to the light press input on input region 456.In some embodiments, the device generates a failure tactile output(e.g., a whole device failure buzz 532 as opposed to localized tactileoutput at input region 456) to indicate that no further change can bemade to the volume level in response to the light press input on inputregion 456, as shown in FIG. 5B6.

FIG. 5B7 illustrates that, another input by contact 534 is detected oninput region 457 (e.g., the input region currently corresponds to avolume down function of the device) after termination of the input bycontact 518 or the input by contact 530. In response to the input bycontact 534 on input region 457, the device displays volume control 520without modifying the volume level when the intensity of the input bycontact 534 has not exceeded the first intensity threshold, anddecreases the volume level by a respective amount each time theintensity of the input by contact 534 increases above the firstintensity threshold. In some embodiments, the device displays visualfeedback (e.g., an animated arrow pointing downward) that indicates thedirection in which the device attribute will be changed if a press inputis detected on the input region 457 that is currently touched by theuser. In FIG. 5B7, the device decreased the current device volume inresponse to detecting the increase of the intensity of contact 534 abovethe first intensity threshold. The device further generates a localizedtactile output 536 at input region 457 to indicate that the criteria fordeceasing the volume level has been met by the input, and that thevolume level has been decreased accordingly. In some embodiments, thetactile output profile for localized tactile output 536 is differentfrom the tactile output profile for localized tactile output 524 in FIG.5B3, to reflect the difference in the direction of the changes in thevolume level.

FIGS. 5B8-5B9 illustrates that, after a decrease in intensity of theinput by contact 534 on input region 457 (as shown in FIG. 5B8), theintensity of the input by contact 534 increases above a second intensitythreshold (e.g., a deep press intensity threshold IT_(D)). In responseto detecting that the intensity of the input by contact 534 has exceededthe second intensity threshold, the device resets the volume level ofthe device to a preset value corresponding to input region 457 (e.g., aminimum value corresponding to muting the volume of the device), asshown in FIG. 5B9. In conjunction with resetting the volume level tomute, the device also generates a localized tactile output (e.g.,tactile output 538 with a tactile output profile that is different fromthe tactile output profiles for tactile outputs 524, 536, and 526) atinput region 457 to indicate that a minimum value of the volume levelhas been reached. In some embodiments, the device also generates a wholedevice tactile output 540 in conjunction with localized tactile output538 to indicate that the volume has been muted in response to the deeppress input by contact 534. FIG. 5B9 illustrates the scenario where theinput is detected on the input region 457 that is used as volume downbutton. If the contact 534 had been detected on input region 456 (e.g.,the input region used as the volume up button) instead, the device wouldreset the volume level of the device to a preset value corresponding toinput region 456 (e.g., a maximum volume level), in response todetecting the intensity of the input exceeding the second intensitythreshold, and the device would generate corresponding localized tactileoutput at input region 456 and whole device tactile output to indicatethe abrupt resetting of the volume level. In some embodiments, inaddition to or in lieu of a requirement that the intensity of thecontact increases above the deep press intensity threshold in order toreset the volume level to a preset maximum or minimum value, the devicerequires that the contact intensity be maintained above the light pressintensity threshold for at least a threshold amount of time T in orderto reset the volume level to a preset maximum or minimum value.

FIGS. 5B10 and 5B11 illustrate that the same set of input regions 456and 457 correspond to multiple device parameters (e.g., including atleast the volume level setting and the screen brightness setting). Thedevice toggles or switches the currently selected device parameter ofthe input regions to a next device parameter in the multiple deviceparameters in response to simultaneous press inputs detected on bothinput regions 546 and 457. In FIG. 5B10, after lift-off of contact 534from input region 457 and upon initial touch down of two contacts 542and 544 on input regions 456 and 457 (e.g., intensities of both contacts542 and 544 are below the first intensity threshold (e.g., the lightpress intensity threshold IT_(L))), respectively, the device redisplaysvolume control 520 on the display to indicate the currently selecteddevice parameter for the input regions, and to indicate the currentvalue of the device volume setting. In FIG. 5B11, upon detecting anincrease in intensity of both contacts 542 and 544 above the firstintensity threshold (e.g., the light press intensity threshold IT_(L)),the device switches the currently selected device parametercorresponding to the input regions 456 and 457 to the display brightnesssetting. In response to the simultaneous light press inputs by contact542 and 544 on input regions 456 and 457, the device replaces thedisplay of volume control 520 with display of brightness control 550.Brightness control 550 is a slider control with an indicator indicatingthe current brightness setting of the display. In conjunction withreplacing display of volume control 520 with brightness control 550, thedevice generates localized tactile output 546 and 548 (e.g., with thesame tactile output profile as each other, and different tactile outputprofile from those of tactile outputs 524, 526, 536, 538) at inputregions 456 and 457, respectively, to indicating that the device hasswitched to another parameter of the multiple device parameters as thecurrently controlled device parameter for the input regions 456 and 457.After termination of the inputs by contacts 542 and 544, the deviceresponds to press inputs on input regions 456 and 457 to adjust thebrightness setting of the device in a manner similar to that describedwith respect to the volume setting in FIGS. 5B1-5B10. In response toanother simultaneous light press inputs on input regions 456 and 457,the device switches the currently controlled device parameter for theinput regions back to the volume setting (e.g., if the input regionsonly correspond to two device parameters) or switches to the next deviceparameter of the multiple device parameters.

FIGS. 5C1-5C20 illustrate exemplary user interfaces for providing hapticand visual feedback for different types of button interactionscorresponding to presenting and interacting with a retractable menu inaccordance with some embodiments. The user interfaces in these figuresare used to illustrate the processes described below, including theprocesses in FIGS. 7A-7E and 12A-12E.

FIG. 5C1 illustrates device 100 displaying a user interface of a browserapplication. Device 100 includes input region 455 (e.g., a power buttondisposed on a peripheral side of the device adjacent to the edge of thedisplay) that is coupled with a localized tactile output generator forgenerating localized tactile outputs at input region 455.

In FIG. 5C2, device 100 detects contact 552 on input region 455. Inresponse to detecting contact 552 (e.g., intensity of contact 552exceeds contact detection intensity threshold IT₀), the device displaysprecursor object 554 corresponding to a retractable menu next to inputregion 455. In some embodiments, precursor object 554 is animated (e.g.,with movement or changing visual characteristic) to prompt the user topress harder on input region 455 (e.g., exerting more pressure in thedirection toward the display). If the user simply rests his/her fingeron input region 455, the device maintains display of precursor object554, without displaying the menu. The device ceases to display precursorobject 554 if lift-off of contact 552 is detected before the menu isrevealed. In some embodiments, the device darkens and blurs thecurrently displayed user interface underneath precursor object 554 tofocus the user's attention on precursor object 554.

FIGS. 5C3-5C5 illustrate that, as intensity of input by contact 552increases and then decreases, the device reveals a menu with multipleselectable options by an amount that changes dynamically with theintensity of contact 552. As shown in FIG. 5C3, when intensity of inputby contact 552 increases, a portion of menu 556 is revealed (e.g.,sliding in onto the display from the side of the display adjacent toinput region 455) showing a first selectable option 558 (e.g., an optionto turn on/off the Do Not Disturb mode of the device). As the intensityof input by contact 552 continues to increase, additional portions ofmenu 556 is revealed (e.g., menu sliding in further in the direction ofthe force/pressure exerted by the user's finger on input region 455)showing a second selectable option 560 (e.g., an option to turn off thedevice). In FIG. 5C5, as the intensity of input by contact 552 decreasesagain, menu 556 retracts toward the edge of the display adjacent toinput region 455. If lift-off of contact 552 is detected, menu 556retracts completely and ceases to be displayed. In some embodiments,before menu 556 is fully expanded to show all available options on themenu, if a tap input (e.g., by another contact on the touch-screen) isdetected on one of the options in the menu that are currently visible onthe display, the selected option will be invoked, and the deviceperforms an operation corresponding to the selected option (e.g., turnon “Do Not Disturb” mode, or turn off the device).

FIGS. 5C6-5C7 illustrate that, if before lift-off of contact 552, thedevice detects that the intensity of the input by contact 552 increasesabove a first intensity threshold (e.g., a light press intensitythreshold IT_(L)), the device fully displays menu 556 and maintainsdisplay of menu 556 after lift-off of contact 552. The fully displayedmenu 556 includes a third selectable option 562 (e.g., an option to usea payment function of the device). In some embodiments, the fullydisplayed menu 556 also includes indicator 5641 to indicate availabilityof additional functions through menu 556 with an enhanced press input(e.g., a deep press input or a long press input on input region 455).Examples of these additional functions are discussed with respect toFIGS. 5G1-5G5 and 5H1-5H5. As shown in FIG. 5C6, in conjunction with thedetection of the light press input by contact 552 and full display ofmenu 556, the device generates localized tactile output 566 at inputregion 455. In some embodiments, localized tactile output 566 issynchronized with intensity of contact 552 reaching the first intensitythreshold. In some embodiments, the menu overshoots and snaps back toits fully opened state after the first intensity threshold is reached,and the device synchronizes the tactile output 566 with the finalsettlement of the menu in its fully open state. FIG. 5C7 illustrates thefully opened state of menu 556 after lift-off of contact 552 has beendetected. Each option in the fully displayed menu 556 is selectivelyactivatable by a tap input at a location on the touch-screencorresponding to the option.

FIGS. 5C8-5C9 illustrate that, a tap input by contact 5661 on the firstselectable option 558 causes the device to activate the “Do Not Disturb”mode of the device. In FIG. 5C9, the device generates a whole devicetactile output (e.g., tactile output 5601) and displays a notification(e.g., notification 5621) to indicate to the user that “Do Not Disturb”mode has been turned on in response to the tap input by contact 5663. Inthis example, option 558 is a toggle button, and the device turns offthe “Do Not Disturb” mode if option 558 is tapped again. Thenotification 5621 ceases to be displayed after a preset time period.

FIGS. 5C10-5C12 illustrate scrolling a selection indicator (e.g.,selection indicator 568) through the multiple selectable options (e.g.,options 558, 560, and 562) to change a currently selected option in menu556. In FIG. 5C10, the device detects contact 564 on input region 455,and downward movement of contact 564 along the longitudinal direction ofthe elongated input region 455. In response to detecting the start ofthe downward swipe input by contact 564 on input region 455 (e.g.,without requiring an intensity of the contact reaching a hint intensitythreshold or a light press intensity threshold), the device displaysselection indicator 568 at a location of the currently selected option(e.g., option 558) in menu 556. The selection indicator may take ondifferent forms (e.g., as a spotlight, a highlight, a cursor, a specialvisual effect applied to a selected option, etc.) in differentembodiments. In FIG. 5C11, as the downward swipe input by contact 564 oninput region 455 continues, the device changes the currently selectedoption to the second option 560 in menu 556, and moves selectionindicator 568 to the location of the second option 560. In FIG. 5C12, asthe downward swipe input by contact 564 on input region 455 continuesfurther, the device changes the currently selected option to the thirdoption 562 in menu 556, and moves selection indicator 568 to thelocation of the third option 562. In FIGS. 5C10-5C12, the swipe input isprovided by a continuously maintained contact 564. The same effect shownin FIGS. 5C10-5C12 can also be achieved using multiple consecutive shortdownward swipes on input region 455. As shown in FIGS. 5C10-5C12, thedevice generates localized tactile output 5663 on input region 455during the swipe input by contact 564. In some embodiments, thelocalized tactile output is focused on the current location of thecontact 564 along input region 455. In some embodiments, the localizedtactile output is a continuous sequence of identical discrete tactileoutputs with fixed durations, simulating a haptic feedback of a roughsurface or the weight of a mechanical scroll wheel. In some embodiments,the switching from one selectable option to the next selectable optionis triggered by the movement of contact 564 meeting preset movementcriteria (e.g., a distance threshold), and the device generates arespective discrete tactile output in conjunction with satisfaction ofthe preset movement criteria and the switching of the currently selectedoption, to simulate a detent produced by a mechanical scroll wheel ordial. Although a downward swipe is used in the example shown in FIGS.5C10-5C12, the device switches the currently selected option in thereverse direction in menu 556, in response to an upward swipe input oninput region 455 and provides corresponding localized tactile outputs atinput region 455.

In FIG. 5C13, while option 562 is the currently selected option in menu556, the device detects a light press input (e.g., an input with anintensity exceeding a hint intensity threshold or light press intensitythreshold) by contact 570 on input region 455. In response to the lightpress input, the device activates the function corresponding to thecurrently selected option 562 (e.g., a payment function). In response todetecting the light press input, the device generates localized tactileoutput 572 to indicate satisfaction of the intensity requirement and theactivation of the selected option. In some embodiments, the device alsohighlights the selected option when it is activated. The tactile output572 indicating activation of the selected option has a distinct tactileoutput profile from the tactile output 566 indicating switching of theselected option and/or the scrolling of the selection indicator.

FIGS. 5C14-5C17 illustrate scrolling through a number of selectableoptions in a menu of payment options (e.g., menu 574 including multipleselectable payment cards 576, 582, 588, etc.) using swipe inputs (e.g.,upward and downward swipes) along the longitudinal direction of theelongated input region 455. As shown in FIG. 5C14, after the paymentoption 562 is activated by a light press input by contact 570, thedevice replaces display of menu 556 with display of menu 574. A defaultpayment card 576 (e.g., the last used payment card) is the currentlyselected option in menu 574 before any additional input is detected. Anenlarged image of the currently selected payment option (e.g., an image578 of the currently selected card 576) is optionally displayed withmenu 574. In FIG. 5C15, the device detects a downward swipe input bycontact 580 along the longitudinal direction of input region 455, and inresponse to detecting the downward swipe input, the device scrollsforward in the list of payment options in menu 574 and selects the nextpayment card 582 in menu 574. The currently selected payment card 582 isenlarged, and an enlarged image 584 of the currently selected card isdisplayed. In FIG. 5C16, the device detects a continuation of thedownward swipe input by contact 580 along the longitudinal direction ofinput region 455, and in response to detecting the continuation of thedownward swipe input, the device scrolls forward in the list of paymentoptions in menu 574 and selects the next payment card 588 in menu 574.The currently selected payment card 588 is enlarged, and an enlargedimage 590 of the currently selected card is displayed. In someembodiments, the list of payment options is implemented as a circularlist, and the user can continue to scroll through the list and return tothe first payment option 574 by one or more additional downward swipeson input region 455. In FIG. 5C17, the device detects an upward swipeinput by contact 580 (e.g., reversal of the downward movement by contact580 along the longitudinal direction of the elongated input region 455).In response to the upward swipe input by contact 580, the device scrollsbackward in the list of payment options in menu 574 and selects theprevious payment card 582 in menu 574 again.

As shown in FIGS. 5C14-5C17, the device generates localized tactileoutput on input region 455 during the swipe inputs by contact 580. Insome embodiments, the localized tactile outputs are focused on thecurrent location of contact 580 along input region 455 (e.g., with mostenergy of each tactile output focused directly under the finger contact,and the focuses of a sequence of tactile outputs moving downward orupward with the finger contact during the downward or upward swipe). Insome embodiments, the localized tactile output is a continuous sequenceof identical discrete tactile outputs with fixed durations, simulating ahaptic feedback of a rough surface or the weight of a mechanical scrollwheel. In some embodiments, the localized tactile output 586 providedduring a downward swipe input and the localized tactile output 592provided during an upward swipe have different tactile output profilesto indicate a difference in the swipe directions. For example, thetactile outputs provided in response to a downward swipe each include apattern with a large peak followed by a small peak; while the tactileoutputs provided in response to an upward swipe each includes a patternwith a small peak followed by a large peak. In some embodiments, theswitching from one selectable option to the next selectable option istriggered by the movement of contact 564 meeting preset movementcriteria (e.g., a distance threshold), and the device generates arespective discrete tactile output in conjunction with satisfaction ofthe preset movement criteria and the switching of the currently selectedoption, to simulate a detent produced by a mechanical scroll wheel ordial. Similarly, the device switches the currently selected option inthe reverse direction in menu 574, in response to an upward swipe inputon input region 455 and provides corresponding discrete localizedtactile outputs (e.g., with a different tactile output profile from thatof the discrete localized tactile outputs 586) at input region 455.

In FIG. 5C18, while payment option 582 is the currently selected paymentoption, a tap input on input region 455 confirms selection of paymentoption 582, e.g., as indicated by further highlighting of the paymentcard image 584 and payment option 582 in menu 574. In some embodiments,a tap input is not necessary (e.g., in the case of actions less criticalthan payment), and the currently selected option is usable directlywithout further confirmation. In some embodiments, the device generateslocalized tactile output 596 to indicate confirmation of the selectionof the payment option 582.

In FIG. 5C19, while the payment option 582 is selected and confirmed,the device is brought into proximity with (e.g., within a presetdistance of) a point of sale device (e.g., POS device 598) (e.g., withthe user holding the device and making contact with the peripheral sidesof the device at input regions 455, 456, and 457). In response todetecting POS device 598 within the preset distance, device 100initiates a countdown (e.g., a countdown corresponding to a presetcancelation period) before the payment operation is executed using thecurrently selected payment card 582. During the countdown, if the devicemoves away from POS device 598, the payment operation is canceled. Ifthe device is maintained within the preset distance of the POS device,the payment operation is executed at the end of the countdown. In someembodiments, if the actual execution of the payment operation takes afinite amount of time and is cancellable within a period of time afterthe execution of the payment operation is stated, the device optionallystarts the execution of the payment operation at a time whencancellation of the payment operation can still be performedsuccessfully before the end of the countdown. As shown in FIG. 5C19, thedevice displays a countdown in the form of a completion meter 593 (e.g.,indicating 70% completion, or 30% remaining of the total cancelationperiod). In some embodiments, the device displays prompt 591 requiringthe user to maintain the device within the preset threshold distance ofthe POS device 598 if the user wishes to complete the payment operation,rather than cancelling it.

In FIG. 5C19, the device provides a sequence of coordinated tactileoutputs 597 during the countdown period where cancellation of theoperation is still permitted. In some embodiments, the coordinatedtactile outputs 597 are generated in repeated sequences around the inputregions 455, 456, and 457. In some embodiments, the amplitudes of thetactile outputs 597 at the different input regions around the peripheralsides of the device indicates a respective direction (e.g., a clockwisedirection, with input region 457 having the lowest tactile outputstrength (e.g., with localized tactile output 597-1), input region 456having the intermediate tactile output strength (e.g., with localizedtactile output 597-2), and input region 455 having the highest tactileoutput strength (e.g., with localized tactile output 597-3)). In someembodiments, the timing of the tactile outputs at the different inputregions are also offset by different amounts from a common referencetime (e.g., with the tactile output (e.g., localized tactile output597-1) at input region 457 having the earliest start time, followed bythe tactile output (e.g., localized tactile output 597-2) at inputregion 456, followed by the tactile output (e.g., localized tactileoutput 597-3) at input region 455). In some embodiments, the timing ofthe tactile outputs at the different input regions around the peripheralsides of the device corresponds to a respective direction (e.g., aclockwise direction, a rightward direction, etc.) relevant to thecurrent usage scenario (e.g., corresponding to the movement direction ofthe countdown indicator). In some embodiments, the manner ofcoordination (e.g., in terms of relative strengths, timing, and tactileoutput patterns) between the individual tactile outputs 597 provided atthe different input regions 455, 456, and 457 is specified in thetactile output profiles of the tactile outputs, and the coordination iscontrolled by a central controller (e.g., haptic feedback controller 161in FIG. 1C). In some embodiments, in addition to the localized tactileoutputs 597, the device also provides a whole device tactile output(e.g., a continuous tactile output (e.g., a continuous buzz)) thatvibrates the whole device during the countdown, to alert the user thatexecution or completion of the payment operation is imminent, if notcanceled in time. In some embodiments, the strength of the whole devicetactile output increases monotonically with the progress of thecountdown indicator to increase the urgency of the warning to the user.

FIG. 5C20 illustrates that, when the countdown is concluded, the deviceexecutes or completes the payment operation, and cancelation of theoperation is no longer available by moving the device away from the POSdevice 598. In some embodiments, the device provides a whole devicetactile output 595 to indicate execution or completion of the paymentoperation. A dismissal affordance 587 is optionally provided on thedisplay to dismiss menu 574.

FIGS. 5D1-5D24 illustrate exemplary user interfaces for providing hapticand visual feedback for different types of button interactionscorresponding to presenting and interacting with a retractable menu(e.g., a clipboard) in accordance with some embodiments. The userinterfaces in these figures are used to illustrate the processesdescribed below, including the processes in FIGS. 7A-7E and 12A-12E.

FIG. 5D1 illustrates a user interface (e.g., user interface 516) of anapplication (e.g., same as that shown in FIG. 5C1). Upon detection ofcontact 585 on input region 455, the device displays precursor object554 that corresponds to retractable menu 556 (e.g., same as that shownin FIG. 5C2), as shown in FIG. 5D2. The device detects movement ofcontact 585 in a direction across the elongated input region 455 (e.g.,from the display-side of the device to the backside of the device), asshown in FIGS. 5D2-5D3. In response to detecting the movement of contact585 in the horizontal direction across the elongated input region 455,the device displays a different precursor object 583 corresponding toanother retractable menu (e.g., a clipboard), as shown in FIG. 5D3. Inaddition, the device provides a localized tactile output 581 (e.g., witha tactile output profile that is different from the tactile outputprofiles of tactile outputs 508, 510, 512, 524, 526, 536, 538, 546, 548,566, 5663, 572, 586, 592, 596, 597, and/or 589) in response to detectingthe movement of contact 585 in the horizontal direction across inputregion 455, to indicate the toggling/switching of the menu currentlycontrolled by the input region 455. In some embodiments, the devicetoggles the precursor object that is displayed (e.g., precursor object554 for menu 556 and precursor object 583 for menu 577) in response to ahorizontal swipe input on input region 455, regardless of the directionof the horizontal swipe input (e.g., from display-side to backside, orfrom backside to display-side) detected across the elongated inputregion 455. This is distinct from the behavior of a physical togglebutton where only one swipe direction is effective for each toggle stateof the button. In some embodiments, the input region 455 corresponds toa circular list of three or more menus, and the device scrolls throughthe respective precursor objects of the three or more menus in responseto multiple horizontal swipes across the input region 455 and provides acorresponding localized tactile output (e.g., a tactile output with thesame tactile output profile as tactile output 581) to indicate theswitching of the precursor object and associated menu.

FIGS. 5D4-5D5 illustrate that, as intensity of the input by contact 579increases, the device reveals increasing areas of menu 577 (e.g., aclipboard of objects such as content items (e.g., images, web links,documents, text, etc.) that now corresponds to input region 455. Menu577 is displayed over user interface 516 without blurring and darkeningof user interface 516 (e.g., in contrast to the case of menu 556 in FIG.5C3). For a clipboard menu, the underlying user interface remains fullyvisible, such that the user can clearly see the objects and possibledrop-off locations in the user interface during a copy and pasteoperation involving object(s) in the clipboard.

In FIGS. 5D6-5D8, while menu 577 remains partially revealed as a resultof sustained contact 579 detected on input region 455 and beforeintensity of contact 579 increases above a first intensity threshold(e.g., the light press intensity threshold IT_(L)), contact 575 isdetected at a location on the touch screen that corresponds to image 573in user interface 516 (e.g., a location outside of menu 577). Inresponse to detecting contact 575 meeting preset selection criteria(e.g., long press criteria based on a time threshold being met bycontact 575, or light press criteria based on an intensity thresholdbeing met by contact 575), the device selects image 573. Then, inaccordance with subsequent movement of contact 575 across thetouch-screen to a location within menu 577, a representation (e.g.,thumbnail 571) of the selected image 573 is dragged into menu 577, asshown in FIG. 5D7.

In FIG. 5D8, after thumbnail 571 is dragged into menu 577, the devicedetects lift-off of contact 575. In response to detecting lift-off ofcontact 575, the device inserts thumbnail 571 into menu 577, and a copyof image 573 becomes an object in the clipboard corresponding to menu577. In some embodiments, the device generates whole device tactileoutput 569 to indicate insertion of image 573 into the clipboard.

In FIG. 5D9, after image 573 is inserted into the clipboard by the dragand drop input by contact 575, the device detects lift-off of contact579 before intensity of contact 579 ever increased above the firstintensity threshold (e.g., the light press intensity threshold IT_(L)).In response to detecting the termination of input by contact 579, thedevice retracts menu 577, and user interface 516 is no longer obscuredby menu 577. In the above process illustrated in FIGS. 5D3-5D9, anobject is added into menu 577 without the menu being fully displayed. Insome embodiments, an object can also be added to the menu by a drag anddrop input when the menu has fully snapped opened due to a prior lightpress input.

In FIG. 5D10, another user interface 567 is displayed, e.g., as a resultof launching another application (e.g., launched from the home screen orfrom an application-switcher user interface that had replaced display ofuser interface 516 in FIG. 5D9).

In FIGS. 5D11-5D12, an input by contact 565 is detected on input region455. Upon detection of contact 565, precursor object 583 correspondingto menu 577 is displayed next to input region 455 (as shown in FIG.5D11). When intensity of contact 565 increases, menu 577 graduallyslides onto the display in the direction of the force/pressure exertedby the user's finger on input region 455. Thumbnail image 571 appears asthe first item displayed in menu 577, as shown in FIG. 5D12.

FIGS. 5D13-5D14 illustrate that a drag input by contact 563 has selectedand dragged a copy of thumbnail image 571 onto user interface 567. Whilecontact 563 is maintained on touch screen 112 holding the copy ofthumbnail image 571, lift-off of contact 565 is detected. In response todetecting lift-off of contact 565 from input region 455, the deviceceases to display menu 577, while contact 563 continues to hold onto thecopy of thumbnail image 571 over user interface 567.

In FIG. 5D15, when lift-off of contact 563 is detected, imagecorresponding to thumbnail 571 is dropped into a drop-off locationselected in accordance with the lift-off position of contact 563 andavailable drop-off positions on user interface 567. In some embodiments,the device generates a whole device tactile output 559 to indicatesettlement of the image into its final position on user interface 567.

FIGS. 5D16-5D17 illustrate that, a press input by contact 557 (e.g.,with contact intensity exceeding the first intensity threshold (e.g., alight press intensity threshold IT_(L))) is detected on input region455, and as a result, the device fully expands menu 577 over userinterface 567. When intensity of contact 557 exceeded the firstintensity threshold and the menu is snapped open, the device generateslocalized tactile output 553 to indicate that menu-display criteria aremet and that the menu display will be maintained after lift-off ofcontact 557 from input region 455. When menu 577 is fully open, scrollindicator 555 is displayed in menu 577 to indicate that additional itemsare in menu 577 which can be revealed in response to a scroll input.

FIGS. 5D18-5D19 illustrates that, after menu 577 is fully opened, twoconsecutive downward swipe inputs by contact 551 and 547, respectively,are detected on input region 455. In response to the downward swipeinputs detected on input region 455, menu 577 is scrolled in a directioncorresponding to the direction of the swipe input (e.g., scrolling tothe right when the swipe is downward, and scrolling to the left when theswipe is upward), as indicated by the movement of scroll indicator 555.In addition to the visual feedback, the device also generates localizedtactile outputs 549 and 545, respectively, on input region 455, inresponse to detecting swipe inputs by contacts 551 and 547. Thelocalized tactile outputs 549 and 545 have the same tactile outputprofile which is designed to simulate rotation of a scroll wheel inresponse to the swipe inputs and/or the roughness of the input regionunder the finger contact. In some embodiments, a selection indicatormoves forward and backward through the list of selectable options (e.g.,content items) in menu 577 to indicate a currently selected option inresponse to downward and upward swipe inputs detected on input region455, and a tap input on input region activates the currently selectedoption indicated by the selection indicator.

In FIGS. 5D20-5D22, while menu 577 remains open after having beenscrolled, the device detects contact 543 at a location corresponding toan object 541 in menu 577. In accordance with selection and dragging ofobject 541 by contact 543 across an edge of touch-screen 112 (e.g.,outside of a permissible on-screen drop-off location), the deviceremoves object 541 from menu 577. Upon lift-off of contact 543, thedevice generates whole device tactile output 539 to indicate thatdeletion of object 541 from menu 577 has been completed.

In FIGS. 5D23-5D24, a tap input by contact 537 is detected outside ofmenu 577, and the device dismisses menu 577 from over user interface567.

FIGS. 5E1-5E5 illustrate exemplary user interfaces for providing hapticand visual feedback for guiding device movement to perform a respectivetask when the device is in a media capture mode of operation inaccordance with some embodiments. The user interfaces in these figuresare used to illustrate the processes described below, including theprocesses in FIGS. 9A-9C and 12A-12E.

In FIG. 5E1, user interface 535 is a camera user interface in a panoramacapturing mode. The device detects presence of user's contact (e.g.,contact 527 on the left side of the device, and contact 525 on the rightside of the device) on all three input regions 455, 456, and 457 (e.g.,the device is held in a user's hand, with the user making contact withinput regions 455, 456, and 457). Although simultaneous contact with allinput regions are shown in FIGS. 5E1-5E5, localized tactile outputs aregenerated on two or more of the input regions at various times duringthe process illustrated herein, even if contact is not maintained at alltimes with all input regions during the process. In addition, generationof tactile outputs at input regions 455, 456, and 457, in some cases,are in response to criteria being met by objects detected in the fieldof view of the camera, do not require touch inputs to be detected oninput regions 455, 456, 457.

As shown in FIG. 5E1, at the beginning of the panorama capturingprocess, panel 533 displays a portion of the panorama image that hasalready been captured (e.g., partial image 529) and movement prompt 531(e.g., an arrow) pointing in the direction of device movement requiredto complete the panorama image. Underneath panel 533, the currentlyfield of view of the camera is displayed. The field of view of thecamera is continuously updated during the panorama capturing based onthe current location of the device relative to the outside environment.

In FIG. 5E2, as the device is moved rightward and the panorama capturingis in progress, the device detects that the device is deviating fromhorizontal movement, and the movement of the device needs to shiftupward (e.g., the arrow is below the center line of panel 533), in orderfor the content of the panorama image captured at different pointsduring movement of the device to line up properly (e.g., the top of themountain in consecutive frames of the camera's image stream shouldoccupy the same vertical location in the frames). In accordance with adetermination that the device has deviated from horizontal movement, andthat the device needs to be shifted upward to return objects in thefield of view into proper positions for continuing the panorama imagecapturing, the device generates a coordinated tactile output (e.g.,coordinated tactile output 523) using localized tactile outputgenerators at input regions 456 and 457 on the left side of the device.In accordance with an extended tactile output profile for coordinatedtactile output 523, the localized tactile output generator at inputregion 457 (e.g., the lower input region on the left side of the device)generates a sequence of discrete tactile outputs 523-1 with a firstamplitude (e.g., a lower amplitude) and a gap between consecutive onesof the discrete tactile outputs 523-1 (e.g., as indicated by a series ofsmall circular objects propagating from the lower input region, witheach small circular object representing a respective discrete tactileoutput and with a distance of each respective small circular object fromthe lower input region corresponding an amount of time that has elapsedsince a respective discrete tactile output represented by the respectivesmall circular object was generated). Further in accordance with anextended tactile output profile for coordinated tactile output 523, thelocalized tactile output generator at input region 456 (e.g., the higherinput region on the left side of the device) generates a sequence ofdiscrete tactile outputs 523-2 with a second amplitude (e.g., a higheramplitude) and a gap between consecutive ones of the discrete tactileoutputs (e.g., as indicated by a series of large circular objectspropagating from the higher input region, with each large circularobject representing a respective discrete tactile output and with adistance of each respective large circular object from the higher inputregion corresponding an amount of time that has elapsed since arespective discrete tactile output represented by the respective largecircular object was generated). Further, in accordance with the extendedtactile output profile, the device offsets the timing of localizedtactile outputs 523-1 and localized tactile outputs 523-2 generated atthe lower input region 457 and the higher input region 456, such that acoordinated sequence of discrete tactile outputs 523 are generated atthe two input regions. For example, an example coordinated sequence is“2_(L)-5_(H)-2_(L)-5_(H)-2_(L)-5_(H)- . . . ”, where “2_(L)” representsthe lower strength (e.g., amplitude=2) tactile output generated at thelower input region 457, and “514” represents the higher strength tactileoutput (e.g., amplitude=5) generated at the higher input region 456.Based on the coordinated tactile output sequence 523, the device promptsthe user to shift the device upward (e.g., in the direction going fromthe region with the weaker tactile output 523-1 to the region with thestronger tactile output 523-2). In some embodiments, the coordinatedtactile output sequence is continuously generated at input regions 456and 457 until the device ceases to detect the deviation in the downwarddirection (e.g., until the user has moved the device upward and therightward horizontal movement of the device is restored).

In FIG. 5E3, as the device continues to move rightward and the panoramacapturing progresses toward completion, the device detects that thedevice is deviating from horizontal movement, and the movement of thedevice needs to shift downward (e.g., the arrow is above the center lineof panel 533), in order for the content of the panorama image capturedat different points during movement of the device to line up properly(e.g., the top of the mountain in consecutive frames of the camera'simage stream should occupy the same vertical location in the frames). Inaccordance with a determination that the device has deviated fromhorizontal movement, and that the device needs to be shifted upward toreturn objects in the field of view into proper positions for continuingthe panorama image capturing, the device generates a coordinated tactileoutput (e.g., coordinated tactile output 521) using localized tactileoutput generators at input regions 456 and 457 on the left side of thedevice. In accordance with an extended tactile output profile for thecoordinated tactile output 521, the localized tactile output generatorat input region 457 (e.g., the lower input region on the left side ofthe device) generates a sequence of discrete tactile outputs 521-1 witha third amplitude (e.g., a higher amplitude) and a gap betweenconsecutive ones of the discrete tactile outputs 521-1 (e.g., asindicated by a series of large circular objects propagating from thelower input region, with each large circular object representing arespective discrete tactile output and with a distance of eachrespective large circular object from the lower input regioncorresponding an amount of time that has elapsed since a respectivediscrete tactile output represented by the respective large circularobject was generated). Further in accordance with an extended tactileoutput profile for the coordinated tactile output 521, the localizedtactile output generator at input region 456 (e.g., the higher inputregion on the left side of the device) generates a sequence of discretetactile outputs 521-2 with a fourth amplitude (e.g., a lower amplitude)and a gap between consecutive ones of the discrete tactile outputs(e.g., as indicated by a series of small circular objects propagatingfrom the higher input region, with each small circular objectrepresenting a respective discrete tactile output and with a distance ofeach respective small circular object from the higher input regioncorresponding an amount of time that has elapsed since a respectivediscrete tactile output represented by the respective small circularobject was generated). Further in accordance with the extended tactileoutput profile, the device offsets the timing of the localized tactileoutputs 521-1 and the localized tactile outputs 521-2 generated at thelower input region 457 and the higher input region 456, such that acoordinated sequence of discrete tactile outputs are generated at thetwo input regions. For example, an example tactile output sequence is“2_(H)-5_(L)-2_(H)-5_(L)-2_(H)-5_(L) . . . ”, where “2_(H)” representsthe lower strength (e.g., amplitude=2) tactile output generated at thehigher input region 456, and “5_(L)” represents the higher strengthtactile output (e.g., amplitude=5) generated at the lower input region457. Based on the coordinated tactile output sequence 521, the deviceprompts the user to shift the device downward (e.g., in the directiongoing from the region with the weaker tactile output 521-2 to the regionwith the stronger tactile output 521-1). In some embodiments, thecoordinated tactile output sequence is continuously generated at inputregions 456 and 457 until the device ceases to detect the deviation inthe upward direction (e.g., until the user has moved the device downwardand the rightward horizontal movement of the device is restored).

In FIG. 5E4, as the device continues to move rightward and the devicedetects that panorama capturing is close to completion (e.g., thehorizontal movement of the device meets a predefined near-completionlinear or angular displacement threshold). The device detects that thehorizontal movement is stalled and/or that the device needs to movefurther rightward in order to complete the panorama image. In accordancewith a determination that the device has deviated required rightwardmovement, and that the device needs to be moved rightward to continuingthe panorama image capturing, the device generates a coordinated tactileoutput (e.g., coordinated tactile output 519) using localized tactileoutput generators at input region 456 on the left side of the device andinput region 455 on the right side of the device. In accordance with anextended tactile output profile for coordinated tactile output 519, thelocalized tactile output generator at input region 456 (e.g., the upperinput region on the left side of the device) generates a sequence ofdiscrete tactile outputs 519-1 with a fifth amplitude (e.g., a loweramplitude) and a gap between consecutive ones of the discrete tactileoutputs 519-1 (e.g., as indicated by a series of small circular objectspropagating from the input region 456 on the left side of the device,with each small circular object representing a respective discretetactile output and with a distance of each respective small circularobject from the input region 456 corresponding an amount of time thathas elapsed since a respective discrete tactile output represented bythe respective small circular object was generated). Further inaccordance with an extended tactile output profile for coordinatedtactile output 519, the localized tactile output generator at inputregion 455 (e.g., the input region on the right side of the device)generates a sequence of discrete tactile outputs 519-2 with a sixthamplitude (e.g., a higher amplitude) and a gap between consecutive onesof the discrete tactile outputs (e.g., as indicated by a series of largecircular objects propagating from the right input region 455, with eachlarge circular object representing a respective discrete tactile outputand with a distance of each respective large circular object from theright input region 455 corresponding an amount of time that has elapsedsince a respective discrete tactile output represented by the respectivelarger circular object was generated). Further, in accordance with theextended tactile output profile, the device offsets the timing oflocalized tactile outputs 519-1 and localized tactile outputs 519-2generated at the left input region 456 and the right input region 455,such that a coordinated sequence of discrete tactile outputs isgenerated at the two input regions. For example, an example tactileoutput sequence is “2_(L)-5_(R)-2_(L)-5_(R)-2_(L)-5_(R) . . . ”, where“2_(L)” represents the lower strength (e.g., amplitude=2) tactile outputgenerated at the left input region 456, and “5_(R)” represents thehigher strength tactile output (e.g., amplitude=5) generated at theright input region 455. Based on the coordinated tactile output sequence519, the device prompts the user to shift the device rightward (e.g., inthe direction going from the region with the weaker tactile output 519-1to the region with the stronger tactile output 519-2). In someembodiments, the coordinated tactile output sequence is continuouslygenerated at input regions 456 and 455 until the device's rightwardmotion is restored and/or the panorama is completed.

In FIG. 5E5, once the device has been moved rightward by the requiredamount, panorama capturing is completed. In response to detectingcompletion of the panorama image, the device ceases to display panel 533over the field of view of the camera in the camera user interface 535.The panorama image 515 is saved into the photo library. In addition, thedevice optionally generates localized tactile output 517 (e.g., not partof the coordinated tactile output 519) at input region 455 to indicatethat panorama capturing is completed.

FIGS. 5F1-5F6 illustrate exemplary user interfaces for providing hapticand visual feedback for guiding device movement to perform a respectivetask when the device is in a media capture mode of operation inaccordance with some embodiments. The user interfaces in these figuresare used to illustrate the processes described below, including theprocesses in FIGS. 9A-9C and 12A-12E.

In FIG. 5F1, a camera user interface 513 is displayed, with a field ofview of the camera capturing three subjects dancing (e.g., a man, awoman, and a girl). In FIG. 5F2, while the user holds the device (e.g.,with contacts 511 and 509 on input regions 455, 456, and 457), thedevice detects a light press input by contact 509 (e.g., contactintensity exceeding a first intensity threshold (e.g., a hint intensitythreshold IT_(H))) on input region 455 (e.g., also serving as a powerbutton of the device). In response to detecting the light press input bycontact 509 on input region 455, the device initiates a face detectionand/or autofocus function of the camera application, and displays threeindicators 505-1, 505-2, 505-3 around the faces of the three subjectsthat have been identified in the field of view of the camera. Inaddition, in conjunction with the face detection and autofocus functionbeing activated, the device generates localized tactile output 507(e.g., a series of weak taps) at input region 455 to indicate thatautofocus and face detection has been triggered and ongoing as long asthe intensity of contact 509 is maintained above the first intensitythreshold (e.g., the hint intensity threshold IT_(H)).

In FIG. 5F3, the device detects an increase in intensity of contact 509above a second intensity threshold (e.g., the light press intensitythreshold IT_(L)) that is greater than the first intensity threshold(e.g., the hint intensity threshold IT_(H)). In response to detectingthe intensity of contact 509 increasing above the second intensitythreshold, the device activates a video capturing function of the cameraapplication, e.g., as indicated by the change in button 503. Facedetection and autofocus continue in the field of view of the camera onuser interface 513 as the subjects and the camera move around relativeto one another. In response to detecting the intensity of contact 509exceeding the second intensity threshold, the device generates localizedtactile output 625 at input region 455 to indicate activation of theinput region by contact 509 and activation the video capturing functionof the camera as a result.

FIGS. 5F4-5F5 illustrate that, during the video capturing process, thedevice continuously monitors the location of the faces that areidentified in the field of view, and generates a warning when thedetected faces are about to move out of the field of view. Althoughsimultaneous contact (e.g., contact 511 and 509) with all input regionsare shown in FIGS. 5F4-5F6, localized tactile outputs are generated ontwo or more of the input regions at various times during the processillustrated herein, even if contact is not maintained at all times withall input regions during the process. In addition, generation of tactileoutputs at input regions 455, 456, and 457, in some cases, are inresponse to criteria being met by objects detected in the field of viewof the camera, do not require touch inputs to be detected on inputregions 455, 456, 457.

In FIG. 5F4, when the device detects that one of the subjects' face(e.g., the girl's face) has moved to the left edge of the field of viewof the camera, the device generates a coordinated tactile output (e.g.,coordinated tactile output 601) using the lower input region 457 on theleft side of the device and the input region 455 on the right side ofthe device. In accordance with a determination that an object on thelower left side of the field of view is about to go out of frame, andthat the device needs to be shifted leftward to move the object awayfrom the left edge of the field of view to continue the video capturing,the device generates a coordinated tactile output in accordance with anextended tactile output profile for coordinated tactile output 601. Inaccordance with the extended tactile output profile for coordinatedtactile output 601, the localized tactile output generator at inputregion 457 (e.g., the lower input region on the left side of the device)generates a sequence of discrete tactile outputs 601-2 with a firstamplitude (e.g., a higher amplitude) and a gap between consecutive onesof the discrete tactile outputs 601-2 (e.g., as indicated by a series oflarge circular objects propagating from the lower input region 457, witheach large circular object representing a respective discrete tactileoutput and with a distance of each respective large circular object fromthe lower left input region corresponding an amount of time that haselapsed since a respective discrete tactile output represented by therespective large circular object was generated). Further in accordancewith an extended tactile output profile for coordinated tactile output601, the localized tactile output generator at input region 455 (e.g.,the input region on the right side of the device) generates a sequenceof discrete tactile outputs 601-1 with a second amplitude (e.g., a loweramplitude) and a gap between consecutive ones of the discrete tactileoutputs (e.g., as indicated by a series of small circular objectspropagating from the right input region, with each small circular objectrepresenting a respective discrete tactile output and with a distance ofeach respective small circular object from the right input regioncorresponding an amount of time that has elapsed since a respectivediscrete tactile output represented by the respective small circularobject was generated). Further, in accordance with the extended tactileoutput profile, the device offsets the timing of localized tactileoutputs 601-1 and localized tactile outputs 601-2 generated at the rightinput region 455 and the lower left input region 457, such that acoordinated sequence of discrete tactile outputs is generated at the twoinput regions. For example, an example tactile output sequence is“2_(R)-5_(LL)-2_(R)-5_(LL)-2_(R)-5_(LL) . . . ”, where “2_(R)”represents the lower strength (e.g., amplitude=2) tactile outputgenerated at the right input region 455, and “5_(LL)” represents thehigher strength tactile output (e.g., amplitude=5) generated at thelower left input region 457. Based on the coordinated tactile outputsequence 601, the device prompts the user to shift the device leftward(e.g., in the direction going from the region with the weaker tactileoutput 601-1 to the region with the stronger tactile output 601-2). Insome embodiments, the coordinated tactile output sequence iscontinuously generated at input regions 455 and 457 until the devicedetects that the object corresponding to indicator 505-3 is no longernear the left edge of the field of view of the camera (e.g., the objectis completely off the screen, or is moved back in frame) (e.g., due tomovement of the device and/or movement of the object corresponding toindicator 505-3).

In FIG. 5F5, when the device detects that one of the subjects' face(e.g., the woman's face) has moved to the right edge of the field ofview of the camera, the device generates a coordinated tactile output(e.g., coordinated tactile output 603) using the upper input region 456on the left side of the device and the input region 455 on the rightside of the device. In accordance with a determination that an object onthe upper right side of the field of view is about to go out of frame,and that the device needs to be shifted rightward to move the objectaway from the right edge of the field of view to continue the videocapturing, the device generates a coordinated tactile output inaccordance with an extended tactile output profile for the coordinatedtactile output 603. In accordance with the extended tactile outputprofile for coordinated tactile output 603, the localized tactile outputgenerator at input region 455 (e.g., the input region on the right sideof the device) generates a sequence of discrete tactile outputs 603-1with a first amplitude (e.g., a higher amplitude) and a gap betweenconsecutive ones of the discrete tactile outputs 603-1 (e.g., asindicated by a series of large circular objects propagating from theright input region 455, with each large circular object representing arespective discrete tactile output and with a distance of eachrespective large circular object from the right input regioncorresponding an amount of time that has elapsed since a respectivediscrete tactile output represented by the respective large circularobject was generated). Further in accordance with an extended tactileoutput profile for coordinated tactile output 603, the localized tactileoutput generator at input region 456 (e.g., the upper input region onthe left side of the device) generates a sequence of discrete tactileoutputs 603-2 with a second amplitude (e.g., a lower amplitude) and agap between consecutive ones of the discrete tactile outputs (e.g., asindicated by a series of small circular objects propagating from theupper left input region, with each small circular object representing arespective discrete tactile output and with a distance of eachrespective small circular object from the upper left input regioncorresponding an amount of time that has elapsed since a respectivediscrete tactile output represented by the respective small circularobject was generated). Further in accordance with the extended tactileoutput profile, the device offsets the timing of the localized tactileoutputs 603-1 and the localized tactile outputs 603-2 generated at theright input region 455 and the upper left input region 456, such that acoordinated sequence of discrete tactile outputs are generated at thetwo input regions. For example, an example tactile output sequence is“2_(UL)-5_(R)-2_(UL)-5_(R)-2_(UL)-5_(R) . . . ”, where “2_(UL)”represents the lower strength (e.g., amplitude=2) tactile outputgenerated at the upper left input region 456, and “5_(R)” represents thehigher strength tactile output (e.g., amplitude=5) generated at theright input region 455. Based on the coordinated tactile output sequence603, the device prompts the user to shift the device rightward (e.g., inthe direction going from the region with the weaker tactile output 603-2to the region with the stronger tactile output 603-1). In someembodiments, the coordinated tactile output sequence is continuouslygenerated at input regions 455 and 456 until the device detects that theobject corresponding to indicator 505-2 is no longer near the right edgeof the field of view of the camera (e.g., the object is completely offthe screen, or is moved back in frame) (e.g., due to movement of thedevice and/or movement of the object corresponding to indicator 505-2).

FIG. 5F6 illustrates that, when all of the detected subjects are well inframe, video capturing continues (e.g., with user's finger resting onthe input regions 455, 456, and 457) and the device does not generateany coordinated tactile output to prompt movement of the device.

FIGS. 5G1-5G5 illustrate exemplary user interfaces for providing hapticand visual feedback for warning a user of impending performance of anoperation in accordance with some embodiments. The user interfaces inthese figures are used to illustrate the processes described below,including the processes in FIGS. 10A-10C and 12A-12E.

In FIG. 5G1, menu 556 has been displayed in response to a prior lightpress input (e.g., in accordance with a process analogous to that shownin FIGS. 5C1-5C7). As show in FIG. 5G1, in addition to multipleselectable options (e.g., options 558, 560, and 562), indicator 5641 isdisplayed in menu 556 to indicate availability of an additional function(e.g., an emergency calling function) that can be triggered in responseto an enhanced press input (e.g., a press input with an enhancedduration and/or intensity requirement).

In FIG. 5G2, as intensity of contact 605 on input region 455 increasesabove a light press intensity threshold IT_(L) and continues to increasetoward a deep press intensity threshold IT_(D), the device generatesfirst localized tactile output 607 (e.g., a sequence of weak clicks) toindicate that a threshold intensity for triggering an emergency callingfunction of the device is about to be reached. In some embodiments, thestrength of the localized tactile output is dynamically changed inaccordance with a current intensity of the input by contact 605. In someembodiments, the device also changes the appearance of the indicator5641 to indicate that the intensity of contact 605 is increasing towardthe threshold intensity for triggering the emergency calling function.

In FIG. 5G3, the device detects that an intensity of the input bycontact 605 has increased above the deep press intensity threshold ITS(e.g., warning-initiation criteria have been met). In response todetecting that the intensity of the input by contact 605 has exceededthe deep press intensity threshold IT_(D), the device starts a countdownfor the emergency calling function and generates a first coordinatedtactile output (e.g., coordinated tactile output 609) using localizedtactile output generators located at input regions 455, 456, and 457, asa warning to the user that emergency services will be called at the endof the countdown (e.g., the countdown corresponding to a cancelationperiod) if a cancellation input is not received before the end of thecountdown. In some embodiments, the coordinated tactile output 609generated at the different input regions (e.g., localized tactileoutputs 609-1, 609-2, and 609-3) are identical and synchronized. In someembodiments, the localized tactile outputs generated at the differentinput regions are offset in timing, to create a cycle of movement aroundthe device. In some embodiments, the amplitudes of the tactile outputscontinuously increase over time during the countdown period. As shown inFIG. 5G3, after the warning-initiation criteria are met by the increasein intensity of contact 605 above the deep press intensity threshold,notification 611 is displayed to indicate that emergency calling will beperformed at the end of a 5 seconds countdown, and that intensity of theinput needs to be maintained above the deep press intensity thresholdITS during the countdown period (e.g., a combined threshold of anintensity threshold and a time threshold need to be satisfied) in orderto meet the operation-execution criteria to perform the emergencycalling function. If the intensity threshold is not met or if the inputterminates before the end of the countdown period, the emergency callingoperation will not be performed and the warning tactile outputs will bestopped. In some embodiments, the appearance of indicator 5641 ischanged (e.g., flashes or pulsates) to indicate that operation-executioncriteria are about to be met and emergency calling function is about tobe performed. In some embodiments, in addition to the localized tactileoutputs and the visual feedback for indicating the impending performanceof the emergency calling function, the device also generates acontinuous and escalating (e.g., with increasing amplitudes) wholedevice tactile output (e.g., tactile output 613) to further enhance theeffect of the warning.

In FIG. 5G4, when the countdown period is over and theoperation-execution criteria for performing the emergency callingfunction are met (e.g., the combined intensity and duration threshold ismet by the input by contact 605), the device stops generating thewarning feedback (e.g., localized tactile outputs 609, notification 611,and whole device tactile output 613), and makes the call to emergencyservices. As shown in FIG. 5G4, notification 617 indicates that theemergency call is in progress. In some embodiments, the device generatesa whole device tactile output (e.g., tactile output 615) to indicatethat the emergency calling function is performed and in progress. Atthis point, cancellation of the emergency call operation is no longerpermitted.

In FIG. 5G5, the call is connected, and the device ceases to generateany tactile output. Notification 619 is displayed to indicate that thecall is connected, and a call termination affordance (e.g., button 621)is displayed. A tap input by contact 623 on the call terminationaffordance terminates the emergency call, and the user interface isrestored to the state shown in FIG. 5G1.

FIGS. 5H1-5H5 illustrate exemplary user interfaces for providing hapticand visual feedback for warning a user of impending performance of anoperation (e.g., activating a voice-based digital assistant) inaccordance with some embodiments. The user interfaces in these figuresare used to illustrate the processes described below, including theprocesses in FIGS. 10A-10C and 12A-12E.

In FIG. 5H1, a user interface (e.g., wake screen user interface 501) isdisplayed. In FIG. 5H2, menu 556 has been displayed in response to alight press input by contact 627 (e.g., in accordance with a processanalogous to that shown in FIGS. 5C1-5C7). A localized tactile output(e.g., localized tactile output 629) is generated to indicate that thelight press intensity threshold has been met by the input. As show inFIG. 5H2, in addition to multiple selectable options (e.g., options 558,560, and 562), indicator 5641 is displayed in menu 556 to indicateavailability of an additional function (e.g., a digital assistantfunction) that can be triggered in response to an enhanced press input(e.g., a press input with an enhanced duration and/or intensityrequirement).

In FIG. 5H3, when the intensity of the input by contact 627 ismaintained above the light press intensity threshold IT_(L) for at leasta threshold amount of time T, the device generates a coordinated tactileoutput (e.g., tactile output 631) using localized tactile outputgenerators at input regions 455 and 456. In some embodiments, thelocalized tactile output 631-1 and 631-2 are each a sequence of discretetactile outputs, and the timing of the two sequences of discrete tactileoutputs are slightly offset (e.g., by an amount corresponding to the gapbetween two consecutive discrete tactile outputs in one of thesequences) from each other. In some embodiments, in response todetecting that the intensity of the input by contact 627 is maintainedabove the light press intensity threshold IT_(L) for at least athreshold amount of time T, the device starts to gather the computingresources to start the digital assistant.

In some embodiments, before the digital assistant is fully ready toaccept and respond to voice input, the device displays visual feedback633 and 637 (e.g., text greetings (e.g., “What can I help you with?” “Goahead, I am listening . . . ”), an animation (e.g., animated waveforms),etc.) to indicate that the digital assistant is getting ready, as shownin FIGS. 5H3 and 5H4. In FIG. 5H4, before the digital assistant is fullyready to receive voice input and provide voice-based responses, thedevice continues to generate the localized tactile outputs 631 on theinput regions 455 and 456. In some embodiments, the user does not needto maintain the intensity of the input above the light press intensitythreshold IT_(L) after the digital assistant has been triggered and thetactile outputs 631 are generated. In some embodiments, if acancellation input (e.g., lift-off of contact 627) is detected beforethe digital assistant is fully activated and ready to respond to user'svoice input, the device cease to generate the localized tactile outputs631, and restores the user interface 501.

FIG. 5H5, the device detects that no cancellation input is detectedbefore the digital assistant is fully activated and ready to respond touser's voice input, the device cease to generate the localized tactileoutputs 631 and displays the digital assistant user interface 639. Insome embodiments, in response to detecting that the digital assistant isfully activated and ready to respond to user's voice input and that nocancellation input is detected before the digital assistant is fullyactivated and ready to respond to user's voice input, the device ceasesto generate localized tactile outputs 631, and generates whole devicetactile output 641 to indicate that the digital assistant is fully readyto respond to the user's voice input.

FIGS. 5I1-5I6 illustrate providing, for each respective type of buttoninteraction, tactile outputs that have varying tactile output profilesdepending on the presence an interstitial material between the inputregion and the user's finger in accordance with some embodiments. Thesefigures are used to illustrate the processes described below, includingthe processes in FIGS. 11A-11B.

For illustrative purposes, only two types of materials (e.g., materialof case 643, and material of case 645) are used as example interstitialmaterials in FIGS. 5I1-5I6. In actual usage scenarios, many differenttypes of interstitial materials are possible, and each correspond torespective ways of modifying the tactile output profile of a basetactile output (e.g., the tactile output generated when there is nointerstitial material present) to produce consistent (e.g., nearlyidentical) haptic sensations on the user's finger through the differenttypes of interstitial materials.

In some embodiments, depending on the actual material (e.g., thematerial category (e.g., leather, plastic, silicone, fabric, etc.), oractual properties of the material (e.g., elasticity, density, thickness,stiffness, etc.)) that is present between the input region and theuser's finger, the type of input (e.g., the actual intensity of theinput, or whether the input is a finger-rest input, a light press input,a deep press input, a horizontal swipe input, or a vertical swipe inputon the input region) that is detected, and/or the type of tactile output(e.g., the waveform, the amplitude, the frequency, etc.) that isgenerated, the device uses different strategies to modify the tactileoutput profiles of the tactile outputs that are to be generated at thecorresponding input regions with presence of an interstitial material(e.g., a case, or a glove, or clothing). FIGS. 5I1-5I6 illustratequalitative modifications to the tactile output profiles for differentinterstitial materials, different types of inputs, and different typesof tactile outputs. These qualitative modifications are not meant to beexhaustive and may be varied in various embodiments, depending on theactual usage scenario.

In FIG. 5I1, a press input with an intensity exceeding a hint intensitythreshold IT_(H) is detected on input region 455, and the devicegenerates a localized tactile output in response to detecting the pressinput. In the left portion of FIG. 5I1, tactile output 649 is generated.The tactile output profile (e.g., a sequence of two discrete waveforms)of the tactile output 649 is provided in the bottom of the left portionof FIG. 5I1. In response to this press input, the device optionallyperforms an operation (e.g., displaying unread notifications on adarkened wake screen user interface as shown in FIG. 5A4, selecting anitem in a menu as shown in FIG. 5C13, autofocusing on detected faces asshown in FIG. 5F2) in conjunction with generating the localized tactileoutput at input region 455.

In the middle portion of FIG. 5I1, device 100 (e.g., including inputregion 455) is covered by case 643. The device detects the same pressinput provided by the user (e.g., the intensity of the input isoptionally determined while taking into consideration of the presence ofthe interstitial material between the user's finger and the input region455) on the input region 455. In response to detecting the same pressinput, the device performs the same operation regardless of whether aninterstitial material is present. But in accordance with a determinationthat an interstitial material (e.g., material one) is present, thedevice generates localized tactile output 653 with a modified tactileoutput profile. The modified tactile output profile of tactile output653 is shown at the bottom of the middle portion of FIG. 5I1. In thisexample, the amplitude of the two discrete waveforms in the tactileoutput profile are modified (e.g., amplitude is increased by 50%) toovercome the dampening effect of the case 643, and to produce movementon the interstitial material over the input region on that issubstantially similar to the movement of the input region caused by thetactile output 649 without presence of the interstitial material 643.

In the right portion of FIG. 5I1, device 100 (e.g., including inputregion 455) is covered by case 645. The device detects the same pressinput provided by the user (e.g., the intensity of the input isoptionally determined while taking into consideration of the presence ofthe interstitial material between the user's finger and the input region455) on the input region 455. In response to detecting the same pressinput, the device performs the same operation regardless of whether aninterstitial material is present. But in accordance with a determinationthat an interstitial material (e.g., material two) is present, thedevice generates localized tactile output 657 with a modified tactileoutput profile. The modified tactile output profile of tactile output657 is shown at the bottom of the right portion of FIG. 5I1. In thisexample, the amplitude of the two discrete waveforms in the tactileoutput profile are modified (e.g., amplitude is increased by 50%) toovercome the dampening effect of the case 645. In addition, a secondarywaveform is generated with a slight time offset from the originalwaveform is added to the original waveform in the modified tactileoutput profile, to overcome the blurring effect of the fluffy materialof the case. Based on the modified tactile output profile, the tactileoutput 657 produces movement on the interstitial material over the inputregion that is substantially similar to the movement of the input regioncaused by the tactile output 649 without presence of the interstitialmaterial.

In the examples shown in FIG. 5I2, the interstitial materials arewrapped around the user's finger instead of being wrapped around thedevice. In the left portion of FIG. 5I2, there is no interstitialmaterial present between the user's finger and the input region of thedevice. When the device detects a press input by contact 665 with anintensity above the hint intensity threshold, the device generatestactile output 667, with the same tactile output profile as that oftactile output 649 in FIG. 5I1. In the middle portion of FIG. 5I2, thesame interstitial material used in case 643 is wrapped around the user'sfinger. As a result, when the device detects the same press input madeby contact 669 through the interstitial material, the device generates atactile output 671 with a modified tactile output profile to compensatefor damping and distortion of the tactile output due to the presence ofthe interstitial material. In this example, the modified tactile outputprofile is the same as the tactile output profile of tactile output 653that was generated when the same interstitial material was used in thecase 643. In the right portion of FIG. 5I2, the same interstitialmaterial used in case 645 is wrapped around the user's finger. As aresult, when the device detects the same press input made by contact 673through the interstitial material, the device generates a tactile output675 with a modified tactile output profile to compensate for damping anddistortion of the tactile output due to the presence of the interstitialmaterial. In this example, the modified tactile output profile is thesame as the tactile output profile of tactile output 657 that wasgenerated when the same interstitial material was used in the case 645.In some embodiments, the tactile output profile of a tactile output isoptionally modified differently depending on whether the interstitialmaterial is used in a case wrapped around the device or used in a glovewrapped around the user's finger.

In FIG. 5I3, a press input with an intensity exceeding a light pressintensity threshold IT_(L) is detected on input region 455, and thedevice generates a localized tactile output in response to detecting thelight press input. In the left portion of FIG. 5I3, tactile output 679is generated. The tactile output profile (e.g., a sequence of twodiscrete waveforms (e.g., a stronger waveform followed by a weakerwaveform)) of the tactile output 679 is provided in the bottom of theleft portion of FIG. 5I3. In response to this light press input, thedevice optionally performs an operation (e.g., modifies read/unreadstatus of notifications on wake screen user interface as shown in FIG.5A5, modifying a value of a device parameter as shown in FIGS. 5B3 and5B7, switching to a different parameter control as shown in FIG. 5B11,fully displaying a menu as shown in FIGS. 5C6 and 5D17, activating avideo capturing mode as shown in FIG. 5F3, activating a digitalassistant as shown in FIG. 5H3) in conjunction with generating thelocalized tactile output at input region 455.

In the middle portion of FIG. 5I3, device 100 (e.g., including inputregion 455) is covered by case 643. The device detects the same lightpress input provided by the user (e.g., the intensity of the input isoptionally determined while taking into consideration of the presence ofthe interstitial material between the user's finger and the input region455) on the input region 455. In response to detecting the same lightpress input, the device performs the same operation regardless ofwhether an interstitial material is present. But in accordance with adetermination that an interstitial material (e.g., material one) ispresent, the device generates a localized tactile output 683 with amodified tactile output profile. The modified tactile output profile oftactile output 683 is shown at the bottom of the middle portion of FIG.5I3. In this example, the amplitudes of the two discrete waveforms inthe tactile output profile are modified differently (e.g., amplitude ofwaveform one is increased by 50%, amplitude of waveform two is increasedby 40%) to overcome the dampening effect of the case 643, and to producemovement on the interstitial material over the input region on that issubstantially similar to the movement of the input region caused by thetactile output 679 with presence of the interstitial material.

In the right portion of FIG. 5I3, device 100 (e.g., including inputregion 455) is covered by case 645. The device detects the same pressinput provided by the user (e.g., the intensity of the input isoptionally determined while taking into consideration of the presence ofthe interstitial material between the user's finger and the input region455) on the input region 455. In response to detecting the same pressinput, the device performs the same operation regardless of whether aninterstitial material is present. But in accordance with a determinationthat an interstitial material (e.g., material two) is present, thedevice generates localized tactile output 687 with a modified tactileoutput profile. The modified tactile output profile of tactile output687 is shown at the bottom of the right portion of FIG. 5I3. In thisexample, the amplitudes of the two discrete waveforms in the tactileoutput profile are modified differently (e.g., amplitude of the firstwaveform is modified by 45% and the amplitude of the second waveform areboth increased by 40%) to overcome the dampening effect of the case 645.In addition, a secondary waveform is generated with a slight time offsetfrom the original waveform is added to the original waveform in themodified tactile output profile, to overcome the blurring effect of thefluffy material of the case. Based on the modified tactile outputprofile, the tactile output 687 produces movement on the interstitialmaterial over the input region that is substantially similar to themovement of the input region caused by the tactile output 679 withoutpresence of the interstitial material.

In some embodiments, the tactile output profile of a tactile output isoptionally modified the same way regardless of whether the interstitialmaterial is used in a case wrapped around the device or used in a glovewrapped around the user's finger. In some embodiments, the tactileoutput profile of a tactile output is optionally modified differentlydepending on whether the interstitial material is used in a case wrappedaround the device or used in a glove wrapped around the user's finger.

In FIG. 5I4, a press input with an intensity exceeding a deep pressintensity threshold ITS is detected on input region 455, and the devicegenerates a localized tactile output in response to detecting the deeppress input. In the left portion of FIG. 5I4, tactile output 693 isgenerated. The tactile output profile (e.g., a discrete waveform with astronger segment in the middle flanked by two weaker segments) of thetactile output 693 is provided in the bottom of the left portion of FIG.5I4. In response to this deep press input, the device optionallyperforms an operation (e.g., displaying expanded versions ofnotifications on wake screen user interface as shown in FIG. 5A6,resetting a device parameter as shown in FIG. 5B9, starting a warningperiod of making an emergency call as shown in FIG. 5G3) in conjunctionwith generating the localized tactile output at input region 455.

In the middle portion of FIG. 5I4, device 100 (e.g., including inputregion 455) is covered by case 643. The device detects the same deeppress input provided by the user (e.g., the intensity of the input isoptionally determined while taking into consideration of the presence ofthe interstitial material between the user's finger and the input region455) on the input region 455. In response to detecting the same deeppress input, the device performs the same operation regardless ofwhether an interstitial material is present. But in accordance with adetermination that an interstitial material (e.g., material one) ispresent, the device generates localized tactile output 697 with amodified tactile output profile. The modified tactile output profile oftactile output 697 is shown at the bottom of the middle portion of FIG.5I4. In this example, the amplitude and the shape of the waveform in thetactile output profile are modified (e.g., amplitude of the centralportion of the waveform is increased by 60%, amplitude of the sideportions of the waveform is increased by 40%, and a secondary waveformis added to the central portion of the original waveform and is slightlyoffset from the original waveform) to overcome the dampening effect ofthe case 643, and to produce movement on the interstitial material overthe input region on that is substantially similar to the movement of theinput region caused by the tactile output 697 without presence of theinterstitial material.

In the right portion of FIG. 5I4, device 100 (e.g., including inputregion 455) is covered by case 645. The device detects the same deeppress input provided by the user (e.g., the intensity of the input isoptionally determined while taking into consideration of the presence ofthe interstitial material between the user's finger and the input region455) on the input region 455. In response to detecting the same deeppress input, the device performs the same operation regardless ofwhether an interstitial material is present. But in accordance with adetermination that an interstitial material (e.g., material two) ispresent, the device generates localized tactile output 701 with amodified tactile output profile. The modified tactile output profile oftactile output 701 is shown at the bottom of the right portion of FIG.5I4. In this example, the amplitude and the shape of the waveform in thetactile output profile are modified (e.g., amplitude of the firstwaveform is increased by 40% and the amplitude of the second waveform isincreased by 60%, and a secondary waveform is added to the entireoriginal waveform and is slightly offset from the original waveform) toovercome the dampening effect of the case 645. Based on the modifiedtactile output profile, the tactile output 687 produces movement on theinterstitial material over the input region that is substantiallysimilar to the movement of the input region caused by the tactile output693 without presence of the interstitial material.

In some embodiments, the tactile output profile of a tactile output isoptionally modified the same way regardless of whether the interstitialmaterial is used in a case wrapped around the device or used in a glovewrapped around the user's finger. In some embodiments, the tactileoutput profile of a tactile output is optionally modified differentlydepending on whether the interstitial material is used in a case wrappedaround the device or used in a glove wrapped around the user's finger.

In FIG. 5I5, a long press input with an intensity exceeding a lightpress intensity threshold IT_(L) and a duration of at least T isdetected on input region 455, and the device generates a localizedtactile output in response to detecting the long press input. In theleft portion of FIG. 5I5, tactile output 705 is generated. The tactileoutput profile (e.g., a sequence of three discrete waveforms) of thetactile output 705 is provided in the bottom of the left portion of FIG.5I5. In response to this long press input, the device optionallyperforms an operation (e.g., activating a digital assistant as shown inFIG. 5H3) in conjunction with generating the localized tactile output atinput region 455.

In the middle portion of FIG. 5I5, device 100 (e.g., including inputregion 455) is covered by case 643. The device detects the same longpress input provided by the user (e.g., the intensity of the input isoptionally determined while taking into consideration of the presence ofthe interstitial material between the user's finger and the input region455) on the input region 455. In response to detecting the same longpress input, the device performs the same operation regardless ofwhether an interstitial material is present. But in accordance with adetermination that an interstitial material (e.g., material one) ispresent, the device generates localized tactile output 709 with amodified tactile output profile. The modified tactile output profile oftactile output 709 is shown at the bottom of the middle portion of FIG.5I5. In this example, the amplitude of the waveform in the tactileoutput profile are modified (e.g., amplitude of the three discretewaveforms is increased by 50%) to overcome the dampening effect of thecase 643, and to produce movement on the interstitial material over theinput region on that is substantially similar to the movement of theinput region caused by the tactile output 705 without presence of theinterstitial material.

In the right portion of FIG. 5I5, device 100 (e.g., including inputregion 455) is covered by case 645. The device detects the same longpress input provided by the user (e.g., the intensity of the input isoptionally determined while taking into consideration of the presence ofthe interstitial material between the user's finger and the input region455) on the input region 455. In response to detecting the same longpress input, the device performs the same operation regardless ofwhether an interstitial material is present. But in accordance with adetermination that an interstitial material (e.g., material two) ispresent, the device generates a localized tactile output 713 with amodified tactile output profile. The modified tactile output profile oftactile output 713 is shown at the bottom of the right portion of FIG.5I5. In this example, the amplitude and the shape of the waveform in thetactile output profile are modified (e.g., amplitude of the threediscrete waveforms is increased by 50%) to overcome the dampening effectof the case 645. In addition, a secondary sequence of three discretewaveforms with lower amplitude (e.g., 30% strength) is added to theoriginal sequence in the original tactile output profile. Based on themodified tactile output profile, the tactile output 713 producesmovement on the interstitial material over the input region that issubstantially similar to the movement of the input region caused by thetactile output 705 without presence of the interstitial material.

In some embodiments, the tactile output profile of a tactile output isoptionally modified the same way regardless of whether the interstitialmaterial is used in a case wrapped around the device or used in a glovewrapped around the user's finger. In some embodiments, the tactileoutput profile of a tactile output is optionally modified differentlydepending on whether the interstitial material is used in a case wrappedaround the device or used in a glove wrapped around the user's finger.

In FIG. 5I6, a long finger-rest input with an intensity below the hintintensity threshold IT_(H) and a duration of at least T is detected oninput region 455, and the device generates a localized tactile output inresponse to detecting the long finger-rest input. In the left portion ofFIG. 5I6, tactile output 717 is generated. The tactile output profile(e.g., a sequence of three discrete waveforms with increasingamplitudes) of the tactile output 717 is provided in the bottom of theleft portion of FIG. 5I6. In response to this long finger-rest input,the device optionally performs an operation (e.g., checking thepresence/absence of unread notifications without modifying the status ofthe notifications as shown in FIG. 5A3) in conjunction with generatingthe localized tactile output at input region 455.

In the middle portion of FIG. 5I6, device 100 (e.g., including inputregion 455) is covered by case 643. The device detects the same longfinger-rest input provided by the user (e.g., the intensity of the inputis optionally determined while taking into consideration of the presenceof the interstitial material between the user's finger and the inputregion 455) on the input region 455. In response to detecting the samelong finger-rest input, the device performs the same operationregardless of whether an interstitial material is present. But inaccordance with a determination that an interstitial material (e.g.,material one) is present, the device generates localized tactile output721 with a modified tactile output profile. The modified tactile outputprofile of tactile output 721 is shown at the bottom of the middleportion of FIG. 5I6. In this example, the amplitude of the waveform inthe tactile output profile are modified by different amounts (e.g.,amplitudes of the three discrete waveforms are increased by 40%, 50%,and 60%, respectively) to overcome the dampening effect of the case 643,and to produce movement on the interstitial material over the inputregion on that is substantially similar to the movement of the inputregion caused by the tactile output 717 without presence of theinterstitial material.

In the right portion of FIG. 5I6, device 100 (e.g., including inputregion 455) is covered by case 645. The device detects the same longfinger-rest input provided by the user (e.g., the intensity of the inputis optionally determined while taking into consideration of the presenceof the interstitial material between the user's finger and the inputregion 455) on the input region 455. In response to detecting the samelong finger-rest input, the device performs the same operationregardless of whether an interstitial material is present. But inaccordance with a determination that an interstitial material (e.g.,material two) is present, the device generates localized tactile output725 with a modified tactile output profile. The modified tactile outputprofile of tactile output 725 is shown at the bottom of the rightportion of FIG. 5I6. In this example, the amplitude and the shape of thewaveform in the tactile output profile are modified (e.g., amplitudes ofthe three discrete waveforms are increased by 50%, 60%, and 50%,respectively, and the frequencies of the waveforms are doubled) toovercome the dampening effect of the case 645. Based on the modifiedtactile output profile, the tactile output 725 produces movement on theinterstitial material over the input region that is substantiallysimilar to the movement of the input region caused by the tactile output717 without presence of the interstitial material.

In some embodiments, the tactile output profile of a tactile output isoptionally modified the same way regardless of whether the interstitialmaterial is used in a case wrapped around the device or used in a glovewrapped around the user's finger. In some embodiments, the tactileoutput profile of a tactile output is optionally modified differentlydepending on whether the interstitial material is used in a case wrappedaround the device or used in a glove wrapped around the user's finger.

In the examples above, the inputs detected on the input region do notinclude movement of the contacts. However, in various embodiments, whenan interstitial material is present between the user's finger and theinput region of the device, and the device generates a localized tactileoutput in response to a swipe input detected on the input region (e.g.,in the horizontal direction across the input region, or in the verticaldirection along the input region), the device makes correspondingmodifications to tactile output profiles of localized tactile outputsgenerated at the input region. The modifications are determined based onthe characteristics of the original tactile output profile and thecharacteristics of the interstitial material, such that the movement ofthe interstitial material (e.g., a portion of the interstitial materialproximate to or adjacent to the input region) generated by the modifiedtactile output is substantially similar to movement of the input regionon the device generated by the original tactile output without presenceof the interstitial material (e.g., the modified tactile output producesa similar haptic sensation (e.g., as measured by vibration strengths,frequencies, etc.) on the finger through the interstitial material asthat produced by the original tactile output directly on the finger).

FIGS. 6A-6D are flow diagrams illustrating a method 6000 of querying andmodifying a device attribute in accordance with some embodiments.

Method 6000 provides a mechanism for a user to query the deviceregarding a current state of a preset attribute of the device by a touchinput without altering the attribute of the device, while preserving theoption to alter the attribute with a continuation of the current input.In particular, the device generates an output to indicate the currentstate of the preset attribute (e.g., a current volume level, whether theringer is on, whether there are unread notifications, etc.) withoutaltering the current state of the attribute in response to an initialportion of a touch input on an off-display input region of the device(e.g., a side-button region) when device-query criteria are met, andproceeds to alter the current state of the preset attribute when acontinuation of the input meets device-adjustment criteria. Providing alight-weight mechanism for a user to obtain a current state of a deviceattribute without altering the device attribute and allowing the user todirectly alter the device attribute by continuing the same input (e.g.,with an increase in contact intensity) after seeing the current state ofthe device attribute enhances the operability of the device and makesthe user-device interface more efficient (e.g., by helping the user toprovide inputs required to achieve an intended outcome and reducing usermistakes when operating/interacting with the device, and by reducing thenumber of steps to perform an action), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently. In some embodiments, thecurrent state of the device attribute is provided via a tactile outputwithout unnecessarily powering up the display and/or other functions ofthe device, which further reduces power usage and improves the batterylife of the device when the user frequently checks the current state ofthe device attribute without intending to alter the device attribute.

The method 6000 is performed at an electronic device (e.g., device 300,FIG. 3, or portable multifunction device 100, FIGS. 1A and 4A) with adisplay (e.g., a display that is disposed on a front-side of thedevice), and a first input region (e.g., input regions 455, 456, or 457)that is separate from the display (e.g., a depressible button, a solidstate button, or a touch-sensitive region that is configured to respondto inputs in place of a button). In some embodiments, the device is atablet-shaped device that has a flat front-side (e.g., display-side), aflat back-side (e.g., back-side), and one or more flat, beveled, orcurved peripheral-sides (e.g., four narrow flat faces joined by sharpstraight edges, four narrow flat faces joined by rounded edges, or acontinuous curved surface) that join the front and back sides, withsharp or rounded edges. In some embodiments, the display is atouch-screen display and the touch-sensitive surface is on or integratedwith the display. In some embodiments, the display is separate from thetouch-sensitive surface. In some embodiments, the first input region isa touch-sensitive surface that detects intensity of a contact with thetouch-sensitive surface, e.g., based on capacitive and/or resistivetouch sensing technologies. In some embodiments, the first input regionis a touch-sensitive surface that detects presence of a contact and/orpressing of an input object (e.g., a finger) based on force, pressure,and/or vibration sensors coupled to the surface. In some embodiments,the first input region is a surface that is coplanar with the display(e.g., a touch-screen display). In some embodiments, the first inputregion is a surface (e.g., a protruding region, a recessed region, or aregion that is flush with the surrounding surface) that is disposed on aperipheral-side of the device adjacent to the front-side of the device(e.g., a left edge, a right edge, a top edge, or a bottom edge, or alongtwo or more of those edges). In some embodiments, the electronic devicefurther includes one or more first sensors to detect presence and/orintensities of contacts with the display of the device (e.g., capacitiveand/or resistive sensors to detect intensities of a contact on thetouch-screen display), and one or more second sensors to detect presenceand/or intensities of contacts with the first input region of thedevice, (e.g., capacitive, resistive, pressure, force, and/or vibrationsensors to detect presence, activation, and, optionally, intensities ofcontact with the first input region (e.g., the side “button” region)).In some embodiments, the device further includes one or more tactileoutput generators for generating tactile outputs (e.g., button actuatorsfor generating localized tactile outputs at the first input region,and/or one or more internal actuators for generating whole-devicetactile outputs, etc.). In some embodiments, the method 6000 is governedby instructions that are stored in a computer-readable storage medium(e.g., a non-transitory computer readable storage medium) and that areexecuted by one or more processors of the electronic device, such as theone or more processors 122 of device 100 (FIGS. 1A and 4A). For ease ofexplanation, the following describes method 6000 as performed by thedevice 100. Some operations in method 6000 are, optionally, combinedand/or the order of some operations is, optionally, changed.

In accordance with method 6000 and with reference to FIG. 6A, the devicedetects (6002) a first portion of a first input on the first inputregion (e.g., detecting a contact on the first input region (e.g.,detecting touch-down of a contact on the touch-sensitive side button ofthe device)).

In accordance with method 6000 and with reference to FIG. 6A, inresponse to detecting the first portion of the first input (6004): inaccordance with a determination that the first portion of the firstinput meets first criteria (e.g., device-query criteria), wherein thefirst criteria include a set of criteria that are met when the firstinput is detected on the first input region without a characteristicintensity of the first input increasing above a first intensitythreshold (e.g., in accordance with detecting that a characteristicintensity of the contact is above a first intensity threshold (e.g., acontact detection intensity threshold, or a light press intensitythreshold that is above the contact detection intensity threshold andbelow a deep press intensity threshold (or button activation intensitythreshold))), the device provides a first output (e.g., displaying auser interface object on the display (e.g., a notification, a volumeadjuster, etc.) or providing a tactile output to indicate availabilityof a new notification without displaying the notification or activatingthe display) that indicates a current state of a first attribute of thedevice (e.g., a current volume level, whether the ringer is on, whetherthere are unread notifications, etc.) without altering the firstattribute of the device. This is illustrated in FIGS. 5A1-5A3, where thedevice provides tactile output 508 to indicate the presence of unreadnotifications in response to input by contact 506, for example. This isfurther illustrated in FIG. 5A4, where the device provides tactileoutput 508 and displays a shaded version of wake screen user interface501 to indicate the presence of unread notifications in response toinput by contact 506, for example. This is further illustrated in FIG.5B2, where the device displays control 520 and indicator 522 to indicatethe current volume level of the device, for example.

In some embodiments, in response to detecting the first portion of thefirst input by the contact and in accordance with a determination thatthe first portion of the first input by the contact does not meet thefirst criteria, the device forgoes providing the first output. Forexample, when a user rests a finger on the side button or lightlypresses on the side button, the display is lit and a user interfaceobject corresponding to the attribute of the device, e.g., a volumecontrol, a ringer control, or a notification summary for unreadnotifications, is displayed. The attribute of the device has multiplepossible states (e.g., discrete configurations, values, and/or statuses,or values in a continuous range of values) that the device can take onin accordance automated device adjustments, triggered by externalevents, and/or in accordance with user's prior manual adjustments. Insome embodiments, the user interface object is optionally an interactivecontrol object that can react to user inputs, e.g., a swipe input orpress input to adjust a slider or toggle a button shown on the userinterface object. In some embodiments, the user interface object merelydisplays a visual indication of the state of the attribute. In someembodiments, in addition to or instead of the visual indication of thestate of the attribute, the device output a local tactile output at theside button using the local tactile output generators located at theside button, or a whole-device tactile output using the internalactuators located inside of the device. When the first output thatindicates the current state of the first attribute of the device, thecurrent state of the first attribute is not altered, e.g., when thevolume control is initially displayed in response to the first portionof the input, the volume control shows the current volume level withoutaltering the current volume level; when the ringer control is displayed,the current ringer on/off setting is not altered; when the notificationsummary is initially displayed, it shows the number of unreadnotifications and optionally the senders of the notifications withoutcontent of the notifications; when a tactile output is generated toindicate that there are one or more unread notifications, theread/unread status of the unread notifications are not altered and thenumber of unread notifications at the device is not changed. In someembodiments, when the device has at least one unread notification, thedevice generates a tactile output with a first tactile output pattern(e.g., a double buzz, or a buzz with a tactile output profile thatcorresponds to the total number of unread notifications) in response tothe first portion of the input, and when the device has no unreadnotification, the device generates no tactile output, or a tactileoutput with a second tactile output pattern (e.g., a single buzz or alighter buzz). In some embodiments, the tactile output is generatedwithout activating the display if the display is off at the time thatthe first portion of the input is detected.

In accordance with method 6000 and with reference to FIG. 6A, the devicedetects (6006) a second portion of the input, including detecting anincrease in intensity of the first input (and optionally, includingdetecting lift-off of the contact). In accordance with method 6000 andwith reference to FIG. 6A, in response to detecting the second portionof the first input (6008): in accordance with a determination that thesecond portion of the first input meets second criteria (e.g.,device-adjustment criteria) before an end of the first input isdetected, wherein the second criteria include a set of criteria that aremet when the characteristic intensity of the first input increases abovethe first intensity threshold (e.g., the second criteria are met whenthe device detects an increase of the characteristic intensity of thefirst input above the first intensity threshold after presenting thefirst output and/or while continuing to display the first output), thedevice performs a first operation that alters the first attribute of thedevice (e.g., adjusting the current volume, toggling the ringer on/offcontrol, or turning on the display to show the content of the unreadnotifications (and changing the read/unread status of at least one orall of the unread notifications from “unread” to “read” or changing thepresence of unread notifications at the device from YES to NO.), etc.);and in accordance with a determination that the second portion of thefirst input does not meet the second criteria (e.g., the first inputdoes not meet the second criteria when the characteristic intensity ofthe contact remains below the first intensity threshold before the endof the first input is detected), the device forgoes performing the firstoperation that alters the first attribute of the device (and optionally,ceasing to providing the first output, if the first output is stillbeing provided at this time). This is illustrated in FIG. 5A5, forexample, where the device displays the unread notifications on wakesscreen user interface and changes the unread status of the notificationsto read, when the intensity of the press input by contact 506 exceededthe light press intensity threshold IT_(L). This is further illustratedin FIG. 5B3, where the device increases the volume level when theintensity of the press input by contact exceeded the light pressintensity threshold IT_(L), for example. In some embodiments, the secondcriteria (e.g., the device-adjustment criteria) are not based onintensity of the contact.). For example, in some embodiments, the firstcriteria are met when presence of the contact is detected on the firstinput region, when a light press input is detected on the first inputregion, or when a tap input is detected on the first input region, andthe second criteria are met when a touch-hold input is detected on thefirst input region. In some embodiments, the second criteria are met bya separate input that is detected after the first criteria are met bythe first portion of the first input and, optionally, while the firstoutput is still being provided by the device. For example, when thefirst criteria are met by a tap input or a light press input and atactile output is generated to indicate presence of unread notificationswithout switching the device from a display-off state to a display-onstate, a touch-hold input on the side button that is detected aftertermination of the tap input or light press input meets the secondcriteria, and causes the device to turn on the display and display theunread notifications.

In accordance with method 6000 and with reference to FIG. 6A, providingthe first output that indicates the current state of the first attributeof the device without altering the attribute of the device includes(6010): displaying a first user interface object (e.g., a volume slidercontrol, a ringer toggle control, a brightness slider control) that hasa first visual characteristic (e.g., length, width, an indicatorposition on a slider or dial control, an indicator position on a togglecontrol, etc.) that corresponds to the current state of the firstattribute (e.g., the current volume level, the current brightness level,the current ringer on/off setting, etc.). This is illustrated in FIG.5A4, where the presence of unread notification is illustrated by thepresence of notifications on the darkened wake screen user interface.This is further illustrated in FIG. 5B2, where the control 520 andindicator 522 indicates the current volume level of the device, forexample. In some embodiments, the first visual characteristic has afirst value when the current state of the first attribute is a firststate of multiple possible states of the first attribute, and the firstvisual characteristic has a second value when the current state of thefirst attribute is a second state of the multiple possible states thatis distinct from the first state. In some embodiments, the display isoff before the first input is detected, and the first portion of thefirst input causes the screen to switch from a display-off state to adisplay-on state, and the first user interface object to appear on thedisplay (e.g., over the lock screen, or the wake screen, or a dark orblurred screen without other user interface objects. In someembodiments, the current user interface shown on the display is darkenedor blurred when the first user interface appears on top of the darkenedor blurred user interface. Displaying a user interface object with afirst visual characteristic that corresponds to the current state of thefirst attribute enhances the operability of the device (e.g., byoffering visual feedback to reduce user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 6000 and with reference to FIG. 6B, providingthe first output that indicates the current state of the first attributeof the device without altering the attribute of the device includes(6012): generating a first tactile output (e.g., localized tactileoutput 508 in FIGS. 5A3 and 5A4) (e.g., a whole-device tactile outputproduced by an internal actuator of the device, or a localized tactileoutput that is produced by a tactile output generator (e.g.,oscillators) located on/at the side button of the device) that has afirst tactile output profile (e.g., amplitudes, frequency, modulatingfrequency, number of output bursts, characteristics of the accompanyingaudio outputs, etc.) that corresponds to the current state of the firstattribute (e.g., the battery level (e.g., low, medium, high), thepresence of unread notifications (present/none, or the count of unreadnotifications), the presence of unread voicemails (e.g., present/none,or the count of unread voicemails), the ringer on/off state, the muteswitch on/off state, etc.). In some embodiments, the device forgoesoutputting the first tactile output for a predefined one of multiplepossible states of the first attribute (e.g., when the current batterylevel is high, or when there are no unread notifications at the currenttime). In some embodiments, each of the multiple possible states of thefirst attribute has a corresponding tactile output with a respectivetactile output profile. In some embodiments, the first tactile output issynchronized with display of the first user interface object on thedisplay. Generating a tactile output with a tactile output profile thatcorresponds to the current state of the first attribute providesadditional information about a state of the device in the absence of orin addition to audio and visual feedback. Providing improved feedback(e.g., with the first tactile output) enhances the operability of thedevice (e.g., by reducing user mistakes when operating/interacting withthe device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 6000 and with reference to FIG. 6B, providingthe first tactile output includes (6014): in accordance with adetermination that the display is not active when the first portion ofthe first input is detected, generating the first tactile output withoutactivating the display (e.g., in FIG. 5A3, the display is not activatedwhen tactile output 508 is provided). In some embodiments, the deviceoutputs a predefined tactile output (e.g., a “failure” buzz) when thereis no unread notification, and does not turn on the display if thedisplay is not already active when the first portion of the first inputis detected. In some embodiments, in accordance with a determinationthat there is at least one unread notification, the device activates thedisplay to show a badge indicating the count of unread notificationswhile generating the first tactile output to indicate presence of unreadnotifications at the device. Providing device attribute informationusing a tactile output without turning on the display reduces powerusage and improves battery life of the device (e.g., by eliminating theneed to power up the display each time the user checks the deviceattribute) and enhances the operability of the device (e.g., by reducingvisual distraction caused by displayed information, and improving userprivacy by forgoing turning on the display).

In accordance with method 6000 and with reference to FIG. 6B, the deviceincludes (6016) a first tactile output generator that resides locally atthe first input region (e.g., a surface oscillator that rocks back andforth around a fixed pivot underneath the first input region or othertactile output generator that is capable of generating tactile outputsthat are directed specifically toward the first input region) and thatprovides localized tactile outputs, including the first tactile output,at the first input region. In some embodiments, the device furtherincludes a second tactile output generator located within the device(e.g., a mass that is moved back and forth around a central restlocation) and that provide whole-device tactile outputs that createtactile outputs in many different locations on the device at the sametime, such as a tactile output generator that is coupled to a housing ofthe device. In some embodiments, the internal tactile output generatorprovides whole-device vibrations that are sensed by the user's handholding the device. In some embodiments, the whole-device tactileoutputs generated by the internal tactile output generator correspondsto visual feedback that is displayed in response to touch-inputsdetected on the touch-screen display, as opposed to inputs detected onthe side button. In some embodiments, the first tactile output generatorprovides localized vibrations at the side button region but not thedevice as a whole. The localized vibrations have varying amplitudes atdifferent locations on the device with greater amplitudes on the sidebutton region where a user's finger will typically rest while operatingthe side button region and with lower amplitudes at other places outsideof the side button region where a user's hand will not typically restwhile operating the side button region. In some embodiments, thelocalized tactile outputs generated by the first tactile outputgenerator corresponds to visual feedback that is displayed in responseto inputs detected on the first input region (e.g., the side button).Generating a localized tactile output at the first input region with alocalized tactile output generator improves location correlation betweenthe user's input and the haptic feedback received by the user, makingthe feedback more immediate and intuitive to the user. Providingimproved feedback enhances the operability of the device (e.g., byreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 6000 and with reference to FIG. 6B, generatinga first tactile output that has a first tactile output profile thatcorresponds to the current state of the first attribute includes (6018):in accordance with a determination that the current state of the firstattribute is a first state of multiple possible states of the firstattribute (e.g., a normal battery level, a ringer off state, ornon-urgent notifications present, a first number of unreadnotifications, etc.), generating the first tactile output with a firstamplitude or first pattern (e.g., first amplitude and/or firstfrequency, or a first number of repeated outputs); and in accordancewith a determination that the current state of the first attribute is asecond state (e.g., low battery level, ringer on state, urgentnotifications present, a second number of unread notifications, etc.) ofthe multiple possible states that is distinct from the first state,generating the first tactile output with a second amplitude or secondpattern that is distinct from the first amplitude or first pattern(e.g., a higher amplitude than the first amplitude and/or a higherfrequency than the first frequency, or a second number of repeatedoutputs). For example, the amplitude and/or pitch of the tactile outputincreases with increasing values (or alternatively, decreasing values)of the first attribute (e.g., count of unread notifications or unreadvoicemails, importance of unread messages, battery drainage level, etc.)In another example, the number of repeated outputs (buzzes) correspondsto the number of unread notifications. Utilizing an amplitude or patternof the tactile output to convey the current state of the first attributeis a quick and intuitive way to providing information to the user.Providing improved feedback enhances the operability of the device(e.g., by reducing user mistakes when operating/interacting with thedevice), which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 6000 and with reference to FIG. 6B, the firstportion of the first output is (6020) detected in accordance withoutputs of one or more capacitive touch sensors, and the first intensitythreshold is based on a capacitance threshold (e.g., as opposed to athreshold physical movement or deformation of the first input regionunder the touch). Utilizing capacitive touch sensors to determine theintensity of the input and using a capacitance threshold as an intensitythreshold enhance the operability of the device (e.g., by making thedevice easily configurable and adaptable to individual users' usagehabit via software means), which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more comfortably, quickly, and efficiently.

In accordance with method 6000 and with reference to FIG. 6C, the firstportion of the first output is (6022) detected in accordance withoutputs of one or more force sensors, and the first intensity thresholdis based on a force threshold (e.g., as opposed to a threshold physicalmovement or deformation of the first input region in the direction ofthe force input). Utilizing force sensors to determine the intensity ofthe input and using a force threshold as an intensity threshold enhancethe operability of the device (e.g., by making the device easilyconfigurable and adaptable to individual users' usage habit via softwaremeans), which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device morecomfortably, quickly, and efficiently.

In accordance with method 6000 and with reference to FIG. 6C, the firstcriteria further include (6024) a set of criteria that are met when thefirst input is detected on the first input region for more than athreshold amount of time (e.g., with less than a threshold amount ofmovement across the first input region). Utilizing a time threshold as acondition for an input meeting the criteria for querying the device forthe current state of the device attribute enhances the operability ofthe device and makes the user-device interface more efficient (e.g., byavoiding accidental triggering of the device's response, and reducinguser mistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more comfortably, quickly,and efficiently.

In accordance with method 6000 and with reference to FIG. 6C, detectingthe second portion of the first input includes detecting termination ofthe first input (e.g., including detecting lift-off of the contact fromthe side button (e.g., the intensity of contact falls below a predefinedcontact detection threshold)), and wherein the method includes (6026):in response to detecting the termination of the first input and inaccordance with a determination that the second portion of the firstinput does not meet the second criteria before the termination of thefirst input: ceasing to provide the first output that indicates thecurrent state of the first attribute of the device. For example,providing the first output includes displaying a first user interfaceobject on the display (e.g., after first switching on the display), andceasing to provide the first output includes ceasing to display thefirst user interface object (and, optionally, restore the original userinterface that was displayed before the first portion of the firstinput, or turning off the display if the display was switched on).Automatically ceasing to provide the output that indicates the currentstate of the first attribute when the input terminates without meetingthe device-adjustment criteria enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by reducingthe number of inputs required to achieve an intended outcome), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 6000 and with reference to FIG. 6C, the firstattribute of the device includes (6028) a presence/absence of unreadnotifications at the device. Providing the output that indicates thecurrent state of the first attribute without altering the firstattribute of the device includes displaying an indicator of the unreadnotifications (e.g., displaying a badge with a numerical count of theunread notifications, or forgoing display of the badge or displaying adifferent indicator to indicate absence of unread notifications at thedevice). Performing the first operation that alters the first attributeof the device includes: displaying content (e.g., expanded notificationshowing full content of text message) of at least one of the unreadnotifications (e.g., displaying a listing of the unread notificationswith sender name and a summary or excerpt from the notifications); andchanging a read/unread status of at least one of the unreadnotifications (e.g., from “unread” to “read”). This is illustrated inFIGS. 5A1-5A8, for example. In some embodiments, providing the outputthat indicates the current state of the first attribute without alteringthe first attribute of the device includes providing a tactile outputindicating presence/count of the unread notification without alteringthe read/unread statuses of the unread notifications and withoutactivating the display of the device, and performing the first operationincludes activating the display, displaying content from at least one ofthe unread notifications, and changing the read/unread status of the atleast one of the unread notifications. The heuristic for displaying anindicator of presence/absence of unread notifications without alteringthe read/unread attribute of the notifications in response to a firstportion of the input that meets first criteria, and displaying contentand changing the read/unread status of at least one of the notificationsin response to a second portion of the input that meets second criteria,enhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by reducing the number of inputsrequired to achieve an intended result) which, additionally reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 6000 and with reference to FIG. 6D, the firstattribute of the device includes (6030) an adjustable control (e.g., avolume setting, a ringer on/off setting, a mute/unmute setting, abrightness setting, etc.) at the device. Providing the output thatindicates the current state of the first attribute without altering thefirst attribute of the device includes displaying a control objectcorresponding to the adjustable control, wherein the control objectincludes an indicator of a current value of the adjustable control(e.g., a volume slider control that shows the current volume level, aringer toggle control that shows the current on/off setting of theringer, a mute/unmute toggle control that shows the current mute/unmutestate of the control, a brightness dial control that shows the currentbrightness level, etc.). Performing the first operation that alters thefirst attribute of the device includes changing the current value of theadjustable control. This is illustrated in FIGS. 5B1-5B7, for example.In some embodiments, the first input region includes two or moresub-regions, and a direction of change in the current value of theadjustable control is based on which of the two or more sub-regions aretouched. For example, if an upper portion of the first input region istouched, the value is increased; and if the lower portion of the firstinput region, the value of is decreased. The heuristic for displaying acontrol object with an indicator of a current value of the controlwithout altering the current value of the control in response to a firstportion of the input that meets first criteria, and changing the currentvalue of the control in response to a second portion of the input thatmeets second criteria, enhances the operability of the device and makesthe user-device interface more efficient (e.g., by reducing the numberof inputs required to achieve an intended result) which, additionallyreduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIGS. 6A-6D have been described is merely exemplary and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 7000, 8000, 9000, 10000, 11000, and 12000) are also applicablein an analogous manner to method 6000 described above with respect toFIGS. 6A-6D. For example, the contacts, gestures, user interfaceobjects, tactile outputs, audio outputs, intensity thresholds, timethresholds, focus selectors, and animations described above withreference to method 6000 optionally have one or more of thecharacteristics of the contacts, gestures, user interface objects,tactile outputs, audio outputs, intensity thresholds, time thresholds,focus selectors, and animations described herein with reference to othermethods described herein (e.g., methods 7000, 8000, 9000, 10000, 11000,and 12000). For brevity, these details are not repeated here.

FIGS. 7A-7E are flow diagrams illustrating a method 7000 of presentingand interacting with a retractable menu in accordance with someembodiments.

Method 7000 relates to displaying a precursor (e.g., a visual indicator)of a user interface object with multiple selectable options (e.g., amenu) in response to a first portion of a touch input (e.g., with a lowintensity) detected on an off-screen input region, and revealing theuser interface object by a varying amount that corresponds to varyingintensity of the touch input during a second portion of the touch inputonce the intensity of the input has increased above an intensitythreshold. Displaying the precursor object before the intensity reachesthe intensity threshold and revealing varying amount of the userinterface object in accordance with the varying intensity of the inputenhance the operability of the device and make the user-device interfacemore efficient (e.g., by providing dynamic visual feedback that directlycorresponds to the intensity of the user's input, and helping the userto provide inputs required to achieve an intended outcome and reducinguser mistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

The method 7000 is performed at an electronic device (e.g., device 300,FIG. 3, or portable multifunction device 100, FIGS. 1A and 4A) with adisplay (e.g., a display that is disposed on a front-side of thedevice), and a first input region (e.g., input region 455, 456, or 457)that is separate from the display (e.g., a depressible button, a solidstate button, or a touch-sensitive region that is configured to respondto inputs in place of a button). In some embodiments, the first inputregion is a touch-sensitive surface that detects intensity of a contactwith the touch-sensitive surface, e.g., based on capacitive and/orresistive touch sensing technologies. In some embodiments, the firstinput region is a touch-sensitive surface that detects presence of acontact and/or pressing of an input object (e.g., a finger) based onforce, pressure, and/or vibration sensors coupled to the surface. Insome embodiments, the first input region is a surface that is coplanarwith the display (e.g., a touch-screen display). In some embodiments,the first input region is a surface (e.g., a protruding region, arecessed region, or a region that is flush with the surrounding surface)that is disposed on a peripheral-side of the device adjacent to thefront-side of the device (e.g., a left edge, a right edge, a top edge,or a bottom edge, or along two or more of those edges). In someembodiments, the electronic device further includes one or more firstsensors to detect presence and/or intensities of contacts with thedisplay of the device (e.g., capacitive and/or resistive sensors todetect intensities of a contact on the touch-screen display), and one ormore second sensors to detect presence and/or intensities of contactswith the first input region of the device, (e.g., capacitive, resistive,pressure, force, and/or vibration sensors to detect presence,activation, and, optionally, intensities of contact with the first inputregion (e.g., the side “button” region)). In some embodiments, thedevice further includes one or more tactile output generators forgenerating tactile outputs (e.g., button actuators for generatinglocalized tactile outputs at the first input region, and/or one or moreinternal actuators for generating whole-device tactile outputs, etc.).

In accordance with method 7000 and with reference to FIG. 7A, the devicedetects (7002) a first portion of a first input on the first inputregion (e.g., detecting a contact on the first input region (e.g.,detecting touch-down of a contact on the touch-sensitive side button ofthe device)). This is illustrated in FIG. 5C2, where an input by contact552 is detected on input region 455. This is further illustrated in FIG.5D3, where an input by contact 585 is detected on input region 455.

In accordance with method 7000 and with reference to FIG. 7A, inresponse to detecting the first portion of the first input on the firstinput region: in accordance with a determination that the first portionof the first input meets first criteria (e.g., preview-displaycriteria), wherein the first criteria include a set of criteria that aremet when the first input is detected on the first input region without acharacteristic intensity of the first input increasing above a firstintensity threshold (e.g., in accordance with detecting that acharacteristic intensity of the contact is above a first intensitythreshold (e.g., a contact detection intensity threshold, a light pressintensity threshold (or preview intensity threshold) that is above thecontact detection intensity threshold and below a deep press intensitythreshold (or button activation intensity threshold))), the devicedisplays (7004), on the display, a precursor object (e.g., precursorobject 554 in FIG. 5C2 and precursor object 583 in FIG. 5D3) (e.g., apreview or an indicator that does not include selectable options) of auser interface object that includes one or more selectable options(e.g., a menu with two or more selectable options, a clipboard forstoring one or more copied objects, etc.). In some embodiments, inresponse to detecting the first portion of the first input and inaccordance with a determination that the first portion of the firstinput does not meet the first criteria, the device forgoes displayingthe precursor object. In some embodiments, the display is off before thefirst portion of the first input is detected, and the display is turnedon in response to the first portion of the first input being detectedand in accordance with the determination that the first criteria are metby the first portion of the first input. In some embodiments, the firstportion of the first input is detected while a wake screen userinterface or a lock screen user interface is shown on the display. Insome embodiments, the first portion of the first input is detected whilea home screen or springboard user interface is shown on the display. Ahome screen or springboard user interface displays application launchicons corresponding to different applications installed on the device.When a respective application launch icon on the home screen isactivated (e.g., by a tap input), the device launches the applicationthat corresponds to the respective application launch icon. The homescreen or springboard user interface replaces a currently displayed userinterface when a home button (e.g., hardware, solid state, or virtualhome button) of the device is activated (e.g., by a press input), orwhen a predefined home gesture (e.g., an upward swipe gesture that startfrom the bottom edge of the touch-screen) is detected. The home screenor springboard user interface optionally includes multiple pages eachincluding respective application launch icons for a respective subset ofall applications installed on the device. In some embodiments, inadditional to a home screen user interface, the device also provides amini application object user interface which includes mini applicationobjects (e.g., widgets, plug-ins) corresponding to a selected subset ofall applications installed on the device. The mini application objects,distinct from application launch icons, each provide a subset of contentand/or functions available in their corresponding full applications, andoptionally launch the corresponding full applications when activated ina preset manner (e.g., by a tap input). In some embodiments, the firstportion of the first input is detected while a user interface of anapplication is shown on the display. In some embodiments, the precursorobject has the same appearance regardless of what was being shown on thedisplay when the first portion of the first input was detected.

In accordance with method 7000 and with reference to FIG. 7A, whiledisplaying the precursor object (e.g., precursor object 554 in FIG. 5C2and precursor object 583 in FIG. 5D3) on the display, the device detects(7006) a second portion of the first input on first input region (e.g.,input by contact 552 in FIGS. 5C3-5C4, and input by contact 579 in FIGS.5D4-5D5), including detecting an increase in intensity of the firstinput above the first intensity threshold (e.g., the first portion andthe second portion of the first input are provided by the same contacton the first input region with the contact being maintained throughoutthe first and second portion of the first input).

In accordance with method 7000 and with reference to FIG. 7A, inresponse to detecting the increase in intensity of the first inputduring the second portion of the first input, the device reveals (7008)the user interface object on the display by an amount that varies inaccordance with a current intensity of the first input. This isillustrated in FIGS. 5C3-5C5 and FIGS. 5D4-5D6, for example, where themenus 556 and 577 are revealed by a respective amount that isdynamically changed in accordance with a current intensity of the inputon input region 455. In some embodiments, as the intensity of thecontact increases on the first input region (e.g., the side button), thefirst user interface object (e.g., a menu, a clipboard) slides graduallyonto the display from the edge of the display that is adjacent to thefirst input region (e.g., the side button). In some embodiments, as theintensity of the first contact increases on the first input region(e.g., the side button), the first user interface object (e.g., the menuor the clipboard) fades in gradually onto the display. In someembodiments, the amount by which the user interface object is visible onthe display is variable as the intensity of the contact varies (e.g.,increases and/or decreases) before reaching a second intensity thresholdthat is greater than the first intensity threshold. Once the intensityof the contact increases above a second intensity threshold, the userinterface object is fully displayed. In some embodiments, the directionin which the user interface slides onto the display is the same as orparallel to the direction of the force/pressure exerted by the finger onthe first input region during the second portion of the first input.

In accordance with method 7000 and with reference to FIG. 7A, afterdetecting the second portion of the first input, the device detects(7010) a third portion of the first input on the first input region,including detecting a continued increase in intensity of the first input(e.g., while a portion of the user interface object is revealed inresponse to the second portion of the first input). In response todetecting the continued increase in intensity of the first input: inaccordance with a determination that the third portion of the firstinput meets second criteria, wherein the second criteria include a setof criteria that are met when a characteristic intensity of the firstinput increases above a second intensity threshold that is greater thanthe first intensity threshold (e.g., in accordance with a determinationthat the characteristic intensity of the contact increases above asecond intensity threshold that is higher than the first intensitythreshold, and optionally, in accordance with a determination that thefirst contact is maintained substantially stationary (e.g., touch-holdwith less than a threshold amount of movement) for more than a thresholdamount of time (e.g., a long-press time threshold) after the firstcontact is detected) before an end of the first input is detected (e.g.,detecting an end of the first input includes detecting a release inputsuch as a liftoff of the contact from the first input region ordetecting a decrease in a characteristic intensity of the contact thatdecreases below a release intensity threshold): the device displays theuser interface object that includes the one or more selectable optionson the display (e.g., the menu or clipboard snaps open when the secondcriteria are met, and remains displayed in the fully open state afterthe intensity of the first input decreases and the first inputeventually terminates). This is illustrated in FIG. 5C6 and FIG. 5D17,where menus 556 and 577 are fully opened and remain displayed when theintensity of the input on input region 455 exceeded the light pressintensity threshold. In some embodiments, in accordance with adetermination that the third portion of the first input does not meetthe second criteria (e.g., in accordance with a determination that thecharacteristic intensity of the contact remains below the secondintensity threshold before the end of the first input is detected), thedevice forgoes fully displaying the user interface object (and ceases todisplay the precursor object of the user interface object and anycurrently revealed portion of the user interface object upon terminationof the first input). Displaying the menu after the intensity of theinput increases above the second intensity threshold enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide inputs required toachieve an intended outcome, and by reducing the number of inputsrequired to achieve an intended outcome) which, additionally reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 7000 and with reference to FIG. 7B, afterdetecting the third portion of the first input, the device detects(7012) a fourth portion of the first input on the first input region,including detecting a decrease in intensity of the first input (e.g.,detecting the intensity of the first input decreases to or below thefirst intensity threshold or detecting lift-off of the contact): inaccordance with a determination that the second criteria were met beforedetecting the fourth portion of the first input (e.g., the secondcriteria were met by the third portion of the first input), the devicemaintains display of the user interface object (e.g., as shown in FIG.5D17) without altering an amount by which the user interface object iscurrently displayed (e.g., menu or clipboard remains fully open); and inaccordance with a determination that the second criteria were not metbefore detecting the fourth portion of the first input (e.g., the secondcriteria were not met by the third portion of the first input), thedevice decreases an amount by which the user interface object isrevealed on the display in accordance with a current intensity of thefirst input (and optionally, ceasing to display the user interfaceobject (e.g., as shown in FIG. 5D9) when the current intensity of thefirst input drops below the first intensity threshold). Maintainingdisplay of the menu or decreasing the amount by which the menu isrevealed in accordance with a reduction of intensity of the input,depending on whether the second criteria have been met by earlierportions of the input, enhances the operability of the device and makesthe user-device interface more efficient (e.g., by reducing the numberof inputs required to achieve an intended outcome) which, additionallyreduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In accordance with method 7000 and with reference to FIG. 7B, inresponse to detecting the continued increase in intensity of the firstinput: in accordance with a determination that the third portion of thefirst input meets the second criteria: the device generates (7014) afirst tactile output (e.g., a whole-device tactile output produced by aninternal actuator of the device, or a localized tactile output that isproduced by a tactile output generator (e.g., oscillators) located on/atthe side button of the device) (e.g., a tactile output that has a firsttactile output profile (e.g., amplitudes, frequency, modulatingfrequency, number of output bursts, characteristics of the accompanyingaudio outputs, etc.) (e.g., a tactile output that is synchronized withthe full display of the user interface object when the second criteriaare met). This is illustrated in FIG. 5C6 and FIG. 5D17, for example,where the device generates tactile outputs 566 and 553, respectively,when the menus 556 and 577 snapped open in response to the light pressinput detected on input region 455. Generating a tactile output inresponse to satisfaction of the second criteria (which also leads tofull display of the menu) enhances the operability of the device andmakes the user-device interface more efficient (e.g., by providingimproved feedback and reducing user mistakes when operating/interactingwith the device) which, additionally reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 7000 and with reference to FIG. 7B, the deviceincludes (7016) a first tactile output generator that resides locally atthe first input region and that provides localized tactile outputs,including the first tactile output, at the first input region. Forexample, the first tactile output generator is a surface oscillator thatrocks back and forth around a fixed pivot underneath the first inputregion or other tactile output generator that is capable of generatingtactile outputs that are directed specifically toward the first inputregion. In some embodiments, the device further includes a secondtactile output generator located within the device (e.g., a mass that ismoved back and forth around a central rest location) that createstactile outputs in many different locations on the device at the sametime, such as a tactile output generator that is coupled to a housing ofthe device. In some embodiments, the internal tactile output generatorprovides whole-device vibrations that are sensed by the user's handholding the device. In some embodiments, the whole-device tactileoutputs generated by the internal tactile output generator correspondsto visual feedback that is displayed in response to touch-inputsdetected on the touch-screen display, as opposed to inputs detected onthe side button. In some embodiments, the first tactile output generatorprovides localized vibrations at the side button region but not thedevice as a whole. The localized vibrations have varying amplitudes atdifferent locations on the device with greater amplitudes on the sidebutton region where a user's finger will typically rest while operatingthe side button region and with lower amplitudes at other places outsideof the side button region where a user's hand will not typically restwhile operating the side button region. In some embodiments, thelocalized tactile outputs generated by the first tactile outputgenerator corresponds to visual feedback that is displayed in responseto inputs detected on the first input region (e.g., the side button).}{Providing a localized tactile output at the first input region inresponse to satisfaction of the second criteria by an input detected onthe first input region enhances the operability of the device and makesthe user-device interface more efficient (e.g., by providing improvedfeedback and reducing user mistakes when operating/interacting with thedevice) which, additionally reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 7000 and with reference to FIG. 7B, thegeneration of the first tactile output is (7018) synchronized withsatisfaction of the second criteria by the first input. Synchronizinggeneration of the localized tactile output at the first input regionwith satisfaction of the second criteria by an input detected on thefirst input region enhances the operability of the device and makes theuser-device interface more efficient (e.g., by providing improvedfeedback and reducing user mistakes when operating/interacting with thedevice) which, additionally reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 7000 and with reference to FIG. 7B, thegeneration of the first tactile output is (7020) synchronized withinitial full-display of the user interface object on the display.Synchronizing generation of the first tactile output with full-displayof the user interface object on the display after satisfaction of thesecond criteria by the input enhances the operability of the device andmakes the user-device interface more efficient (e.g., by providingimproved feedback and reducing user mistakes when operating/interactingwith the device) which, additionally reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 7000 and with reference to FIG. 7C, the deviceincludes (7022) a touch-sensitive surface that is distinct from thefirst input region (e.g., the touch-sensitive surface is integrated withthe display in a touch-screen display). The method 7000 includes: whiledisplaying the user interface object including the one or moreselectable options on the display (e.g., while the user interface objectis partially and/or fully revealed on the display), detecting a secondinput on the touch-sensitive surface (e.g., a touch input by a contactthat is distinct from the contact on the first input region in the firstinput) at a location that corresponds to a first selectable option ofthe one or more selectable options in the user interface object; and inresponse to detecting the second input: in accordance with adetermination that the second input meets selection criteria (e.g., theselection criteria are met by a tap input on the first selectableoption), selecting the first selectable option (e.g., visuallyindicating selection of the first selectable option by highlighting thefirst selectable option relative to other selectable options in the userinterface object). This is illustrated in FIG. 5D13, where thumbnailimage 571 is selected by contact 563. In addition, options in menu 556is selectable by tap input. After the menu is displayed in response toan input detected on the off-screen input region, another input detectedon the touch-sensitive surface (e.g., the touch screen) is used toselect an icon on the menu. Allowing selection of a selectable option onthe menu using a touch-sensitive surface distinct from the input regionenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by allowing user to interact with thedevice with multiple input interfaces (e.g., without shuffling thedevice between two hands or adjusting the hand grip) and reducing usermistakes when operating/interacting with the device) which, additionallyreduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In accordance with method 7000 and with reference to FIG. 7C, inresponse to detecting the second input: in accordance with adetermination that the second input meets selection-confirmationcriteria (e.g., detecting the second input further includes detectinglift-off of the contact in the second input from the touch-sensitivesurface (e.g., the touch-screen display) from the location thatcorresponds to the first selectable option), the device performs (7024)a first operation that corresponds to the first selectable option. Thisis illustrated in FIGS. 5C8 and 5C9, where the Do Not Disturb functionin menu 556 is activated by a tap input on option 558, for example. Insome embodiments the selection-confirmation criteria are met withrespect to the first selectable option by a tap input on the firstselectable option. In another example, a swipe input across multipleselectable options changes the currently selectable option, and thefinally selected option is activated upon lift-off of the contact fromthe touch-screen display. In some embodiments, the user interface objectis a listing of payment cards, and tapping on a first payment card inthe listing of payment cards or lifting-off while the first payment cardis selected causes the first payment card to be selected for a paymenttransaction. In another example, the user interface object is a menu ofpower-related functions, including shutdown, restart, sleep, turn-offdisplay, turn on low-power mode, etc., and tapping on a firstpower-related function in the menu or lifting-off while the firstpower-related function is selected causes the correspondingpower-related function to be turned on. Performing an operation thatcorresponds to a selected option after lift-off of the input thatselected the option enhances the operability of the device and makes theuser-device interface more efficient (e.g., by reducing the number ofinputs required to achieve an intended result) which, additionallyreduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In accordance with method 7000 and with reference to FIG. 7C, inresponse to detecting the second input: in accordance with adetermination that the second input meets drag criteria (e.g., the dragcriteria are met by a selection of the first selectable option followedby movement of the first selectable option by a tap-hold input on thefirst selectable option followed by a drag input), the device displays(7026) a representation of the first selectable option outside of theuser interface object (e.g., at a location on the display thatcorresponds to a current location of the contact on the touch-sensitivesurface.). This is illustrated in FIGS. 5D13-5D15, where image 571 isdragged from menu 577 and dropped in user interface 567, for example. Insome embodiments, the user interface object is a listing of visualeffects or filters, and dragging a first visual effect or filter awayfrom the user interface object to a portion of the user interfaceoutside of the user interface object shows a preview of effect of thevisual effect or filter (e.g., a preview of a saturation filter on acurrently displayed image), or applies the visual effect or filter to aportion of a displayed image (e.g., a magnifier object magnifies aportion of the displayed image that corresponds to a location of thecontact on the display, or a blur object blurs a portion of thedisplayed image that corresponds to the location of the contact on thedisplay). In another example, a representation of the first selectableoption is an insertion cursor that indicates where the first selectableoption will be inserted if lift-off of the contact with the touch-screenis detected. Dragging the selected option out of the user interfaceobject in accordance with a determination that the input that selectedthe option also meets drag criteria enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byreducing the number of user inputs required to achieve an intendedresult) which, additionally reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 7000 and with reference to FIG. 7C, inresponse to detecting the second input: in accordance with adetermination that the second input meets drop criteria (e.g., the dropcriteria are met by a release of the contact that is detected after atap-hold input on the first selectable option followed by a drag input),the device displays (7028) a copy of the first selectable option outsideof the user interface object. This is illustrated in FIGS. 5D13-5D15,where image 571 is dragged from menu 577 and dropped in user interface567, for example. In some embodiments, the user interface object is aclipboard including one or more copied objects (e.g., images, links,text, etc.). When a first object is dragged out of the user interfaceobject by a touch-hold input followed by a drag input, an insertioncursor or icon that indicates where the first object will be inserted isshown on the user interface, and when lift-off of the contact isdetected and drop criteria are met, a copy of the first object isdisplayed in the user interface at a final location indicated by theinsertion cursor or icon. After the selected option is dragged out ofthe user interface object in accordance with a determination that theinput that selected the option also meets drag criteria, a copy of theselected option (e.g., an image or item) is dropped at a locationoutside of the user interface object in accordance with a determinationthat the input also meets drop criteria. The dropping of the selectedoption when the input meets drop criteria enhances the operability ofthe device and makes the user-device interface more efficient (e.g., byreducing the number of user inputs required to achieve an intendedresult) which, additionally reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 7000 and with reference to FIG. 7C, the secondinput is detected (7030) while the user interface object is partiallyrevealed on the display, without the second criteria being met by thefirst input. This is illustrated in FIG. 5D13, where image 563 isselected before menu 577 is fully open. In some embodiments, objects inthe menu are selectable before the menu is fully opened (e.g., the usercan select or drag an object from the menu when the menu is onlypartially on the display). In some embodiments, objects in the menu arenot selectable before the menu is fully opened. For example, forclipboard, the objects are not selectable before the clipboard is fullyopen because the user needs to be able to see the stored objects fullybefore making a correct selection. In contrast, for the menu ofpower-related functions, the functions are selectable before the menu isfully open, because the menu is relatively fixed and users are familiarwith the functions. Making the selectable options selectable before theuser interface object is fully revealed enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byreducing the time and the number of user inputs required to achieve anintended result) which, additionally reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 7000 and with reference to FIG. 7D, whiledisplaying the user interface object including the one or moreselectable options on the display (e.g., while the user interface objectis partially and/or fully revealed on the display), the device detects(7032) a third input on the first input region (e.g., a touch input by acontact that is distinct from the contact on the first input region inthe first input, or a touch input by a contact that is the samecontinuous contact as the contact on the first input region in the firstinput), wherein the third input includes movement of a contact acrossthe first input region (e.g., a movement in a direction along alongitudinal direction of the first input region (e.g., verticallyupward or downward)). In response to detecting the third input, thedevice scrolls a selection indicator (e.g., selection indicator 568 inFIGS. 5C10-5C12) through the one or more selectable options on thedisplay in accordance with the movement of the contact in the thirdinput (e.g., scroll the selection indicator downward through the listingof selectable options in the menu or clipboard when the movement is adownward swipe along the first input region, and scroll the selectionindicator upward through the listing of selectable options in the menuor clipboard when the movement is an upward swipe along the first inputregion). This is illustrated, for example, in FIGS. 5C10-5C12, where aselection indicator is moved through selectable options in menu 556 inresponse to swipe input by contact 564 detected on input region 455. Insome embodiments, the menu becomes scrollable through a swipe input onthe first input region after the menu is fully open (e.g., in responseto a press input on the first input region). Scrolling a selectionindicator through the selectable options in the user interface object inresponse to a swipe input across the first input region after the userinterface object is fully displayed in response to a press input on thefirst input region enhances the operability of the device and makes theuser-device interface more efficient (e.g., by reducing the number ofselectable options concurrently displayed on screen, and providingadditional selectable options without cluttering the user interface)which, additionally reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 7000 and with reference to FIG. 7D, while theselection indicator is moved to a first selectable option of the one ormore selectable options on the display in accordance with the movementof the contact in the third input on the first input region, the devicedetects (7034) a fourth input on the first input region that meetsselection-confirmation criteria (e.g., the selection-confirmationcriteria are met when lift-off of the contact on the first input regionis detected, or when an increase in a characteristic intensity of thecontact on the first input region above a predefined confirmationintensity threshold (e.g., the second intensity threshold) is detected).In some embodiments, the selection-confirmation criteria are met bydetecting a press input by a contact followed by detecting lift-off ofthe contact from the first input region. This is illustrated in FIG.5C13, for example. In some embodiments, the selection-confirmationcriteria are met by detecting a contact followed by detecting lift-offof the contact from the first input region without requiring thecharacteristic intensity of the contact to increase above the predefinedconfirmation intensity threshold first. In response detecting the fourthinput that meets the selection-confirmation criteria, the deviceperforms an operation corresponding to the first selectable options(e.g., if the first selectable option is a filter, apply the filter to acurrently displayed image. If the first selectable option is apower-related function for shutting down the device, start a process toshut down the device. If the first selectable option is a music track,starting playback of the music track, etc.). Performing an operationthat corresponds to a currently selected option in response to aselection confirmation input (e.g., a press input or lift-off of thepress input) on the first input region enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byreducing user mistakes when operating/interacting with the device)which, additionally reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 7000 and with reference to FIG. 7D, the deviceincludes (7036) a touch-sensitive surface that is distinct from thefirst input region (e.g., the touch-sensitive surface is integrated withthe display in a touch-screen display). The method 7000 includes: whiledisplaying the user interface object including the one or moreselectable options on the display (e.g., while the user interface objectis partially and/or fully revealed on the display), detecting a fifthinput on the touch-sensitive surface, including: detecting touch-down ofa contact on the touch-sensitive surface (e.g., a touch input by acontact that is distinct from the contact on the first input region inthe first input) at a location that corresponds to a second userinterface object that is displayed outside of the user interface object,detecting movement of the contract across the touch-sensitive surface toa location within the user interface object (e.g., the fifth input is adrag gesture that drags the second user interface object from outside ofthe menu or clipboard to the menu or clipboard), and detecting lift-offof the contact from the touch-sensitive surface while the second userinterface is within the user interface object. In response to detectingthe fifth input, the device adds the second user interface object as anew selectable option in the user interface object. This is illustratedin FIGS. 5D5-5D8, where image 571 is added to menu 577 in response toinput by contact 575. Allowing selection an object from outside of theuser interface object (e.g., a menu or clipboard), dragging the selectedobject into the user interface object, and adding the selected objectinto the user interface object as a new selectable option in the userinterface object, in response to an input (e.g., a touch-drag-dropinput) on the touch-sensitive surface (e.g., a touch-screen) that isdistinct from the first input region, enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byreducing the number of inputs required to achieve an intended result)which, additionally reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 7000 and with reference to FIG. 7E, the deviceincludes (7038) a touch-sensitive surface that is distinct from thefirst input region (e.g., the touch-sensitive surface is integrated withthe display in a touch-screen display). The method 7000 includes: whiledisplaying the user interface object including the one or moreselectable options on the display (e.g., while the user interface objectis partially and/or fully revealed on the display), detecting a sixthinput on the touch-sensitive surface, including: detecting touch-down ofa contact on the touch-sensitive surface (e.g., a touch input by acontact that is distinct from the contact on the first input region inthe first input) at a location that corresponds to a respectiveselectable option that is displayed within the user interface object,detecting movement of the contract across the touch-sensitive surface toa location outside of the user interface object (e.g., the sixth inputis a drag gesture that drags the respective selectable option fromwithin the menu or clipboard to outside of the menu or clipboard), anddetecting lift-off of the contact from the touch-sensitive surface whilethe respective selectable option is dragged outside of the userinterface object. In response to detecting the sixth input, the deviceremoves the respective selectable option from the user interface object(e.g., deleted from the user interface object (e.g., dropped onto theuser interface outside of the user interface object when the input meetsobject-move criteria, or cleared without being dropped onto the userinterface outside of the user interface object when the input meetsobject-deletion criteria)). This is illustrated, for example, in FIGS.5D20-5D22, where image 541 is removed from menu 577 in response to inputby contact 543. Dragging a selectable option out of the user interfaceobject to remove or delete it from the user interface object in responseto an input (e.g., a touch-drag-drop input) on the touch-sensitivesurface (e.g., a touch-screen) that is distinct from the first inputregion enhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by reducing the number of inputsrequired to achieve an intended result) which, additionally reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIGS. 7A-7E have been described is merely exemplary and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 6000, 8000, 9000, 10000, 11000, and 12000) are also applicablein an analogous manner to method 7000 described above with respect toFIGS. 7A-7E. For example, the contacts, gestures, user interfaceobjects, tactile outputs, audio outputs, intensity thresholds, timethresholds, focus selectors, and animations described above withreference to method 7000 optionally have one or more of thecharacteristics of the contacts, gestures, user interface objects,tactile outputs, audio outputs, intensity thresholds, time thresholds,focus selectors, and animations described herein with reference to othermethods described herein (e.g., methods 6000, 8000, 9000, 10000, 11000,and 12000). For brevity, these details are not repeated here.

FIGS. 8A-8D are flow diagrams illustrating a method 8000 of modifying adevice parameter in different manners in accordance with someembodiments.

Method 8000 relates to modifying a parameter of the device in one of twodistinct manners (e.g., a first manner and a second manner distinct fromthe first manner) in response to a press input on an off-display inputregion (e.g., a side button region) in accordance with whether the inputis detected on a first sub-region or a second sub-region of the inputregion. Modifying the parameter in different manners in accordance witha determination of which one of two sub-regions of the input region isactivated by the press input enhances the operability of the device andmakes the user-device interface more efficient (e.g., by reducing numberof inputs required to achieve an intended result), which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently. In addition,forgoing modifying the parameter in the different manners in accordancewith a determination of the input on the input region did not meet theintensity threshold (e.g., the input is not a press input) enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

The method 8000 is performed an electronic device (e.g., device 300,FIG. 3, or portable multifunction device 100, FIGS. 1A and 4A) with adisplay, and an input region that is distinct from the display andincludes a first sub-region (e.g., upper input region 456 on the leftside of the device 100) and a second sub-region (e.g., lower inputregion 457 on the left side of the device 100) (e.g., the input regionis a depressible button, a solid state button, or a touch-sensitiveregion that is configured to respond to inputs in place of a button, andthe first sub-region and the second sub-region are an upper region and alower region of the input region). In some embodiments, the firstsub-region and the second sub-region are continuous and connected toeach other. In some embodiments, the first sub-region and the secondsub-region are separate and discrete from each other. In someembodiments, the first sub-region and the second sub-region areconfigured to move independently of each other and/or have sensors todetect inputs independently of each other. In some embodiments, thefirst input region (and, optionally, each of its sub-regions) is atouch-sensitive surface that detects intensity of a contact with thetouch-sensitive surface, e.g., based on capacitive and/or resistivetouch sensing technologies. In some embodiments, the first input region(and, optionally, each of its sub-regions) is a touch-sensitive surfacethat detects presence of a contact and/or pressing of an input object(e.g., a finger) based on force, pressure, and/or vibration sensorscoupled to the surface. In some embodiments, the first input region(and, optionally, each of its sub-regions) is a surface that is coplanarwith the display (e.g., a touch-screen display). In some embodiments,the first input region (and, optionally, each of its sub-regions) is asurface (e.g., a protruding region, a recessed region, or a region thatis flush with the surrounding surface) that is disposed on aperipheral-side of the device adjacent to the front-side of the device(e.g., a left edge, a right edge, a top edge, or a bottom edge, or alongtwo or more of those edges). In some embodiments, the electronic devicefurther includes one or more first sensors to detect presence and/orintensities of contacts with the display of the device (e.g., capacitiveand/or resistive sensors to detect intensities of a contact on thetouch-screen display), and one or more second sensors to detect presenceand/or intensities of contacts with the first input region (and,optionally, each of its sub-regions) of the device, (e.g., capacitive,resistive, pressure, force, and/or vibration sensors to detect presence,activation, and, optionally, intensities of contact with the first inputregion (e.g., the side “button” region)). In some embodiments, thedevice further includes one or more tactile output generators forgenerating tactile outputs (e.g., button actuators for generatinglocalized tactile outputs at the first input region (and, optionally,each of its sub-regions), and/or one or more internal actuators forgenerating whole-device tactile outputs, etc.).

In accordance with method 8000 and with reference to FIG. 8A, the devicedetects (8002) an input on the input region (e.g., input by contact 518on upper input region 456 in FIG. 5B2, or input by contact 534 on lowerinput region 547 in FIG. 5B7) (e.g., detecting a contact on the firstinput region (e.g., detecting touch-down of a contact on thetouch-sensitive side button of the device)).

In accordance with method 8000 and with reference to FIG. 8A, inresponse to detecting the input on the input region (8004), and inaccordance with a determination that the input (e.g., light press inputby contact 518 on the upper input region 456 in FIG. 5B3) meets firstcriteria (e.g., first button activation criteria), wherein the firstcriteria require an increase in intensity of the input above a firstintensity threshold (e.g., a light press intensity threshold that isabove the contact detection intensity threshold and below a deep pressintensity threshold) while the input is detected in the first sub-region(e.g., upper input region 456 in FIG. 5B3) of the input region, in orderfor the first criteria to be met, the device modifies (8006) a firstparameter of the device in a first manner (e.g., increase the volumelevel (e.g., as illustrated in FIG. 5B3), or scroll upward). In responseto detecting the input (e.g., light press input by contact 534 on thelower input region 457 in FIG. 5B7) on the input region (8004), and inaccordance with a determination that the input meets second criteria(e.g., second button activation criteria) that are distinct from thefirst criteria, wherein the second criteria require an increase inintensity of the input above the first intensity threshold while theinput is detected in the second sub-region (e.g., the lower input region457 in FIG. 5B7) of the input region that is distinct from the firstsub-region, in order for the second criteria to be met, the devicemodifies (8008) the first parameter of the device in a second manner(e.g., lower the volume (e.g., as illustrated in FIG. 5B7) or scrolldownward) that is distinct from the first manner. In some embodiments,the first and second criteria require that the contact is maintainedsubstantially stationary (e.g., touch-hold with less than a thresholdamount of movement) for more than a threshold amount of time (e.g., along-press time threshold) after the contact is detected, withoutrequiring the intensity to rise above the light press intensitythreshold. In response to detecting the input on the input region(8004), and in accordance with a determination that the input did notincrease above the first intensity threshold (e.g., the input did notincrease above the first intensity threshold before lift-off of thecontact is detected), the device forgoes (8010) modifying the firstparameter of the device in the first manner and forgoing modifying thefirst parameter of the device in the second manner (e.g., the devicedoes not change the first parameter in any way).

In accordance with method 8000 and with reference to FIG. 8A, the deviceincludes (8012) one or more first tactile output generators that arelocated at the input region (e.g., one or more surface oscillators thatrock back and forth around respective fixed pivots underneath the inputregion) and that provide localized tactile outputs at the input region.The method 8000 includes: in accordance with a determination that theinput (e.g., input by contact 518 on the upper input region 456 in FIG.5B3) meets the first criteria, generating a first tactile output (e.g.,tactile output 524 in FIG. 5B3) with a first tactile output profile; andin accordance with a determination that the input (e.g., input bycontact 534 on lower input region 457 in FIG. 5B7) meets the secondcriteria, generating a second tactile output (e.g., tactile output 536in FIG. 5B7) with a second tactile output profile. In some embodiments,the first tactile output and the second tactile output have the sametactile output profiles and different accompanying audio outputs (e.g.,one with a higher frequency than the other). In some embodiments, thefirst tactile output and the second tactile outputs have distincttactile output profiles (e.g., frequencies, patterns, amplitudes, etc.).In some embodiments, the first tactile output and the second tactileoutput are generated by the same localized tactile output generator atthe input region (e.g., the side button region), and the tactile outputsare sensed on the input region at any location that the user is makingcontact with the input region. In some embodiments, separate localizedtactile output generators are used to generate localized tactile outputsat each of the first sub-region and the second sub-region of the inputregion. In some embodiments, in accordance with a determination that theinput did not increase above the first intensity threshold, the deviceforgoes generating the first and second tactile outputs. In someembodiments, the device generates a tactile output to indicate thelocations of the first sub-region and the second sub-region to the user.For example, a third tactile output is generated by the first tactileoutput generator when the user's finger is resting on the firstsub-region, and a fourth tactile output is generated by the firsttactile output generator when the user's finger is resting on the secondsub-region, where the third tactile output and the fourth tactile outputhave different tactile output patterns from each other, and from thefirst and second tactile outputs. Providing a localized tactile outputin the input region where the input is detected, when the criteria formodifying the parameter are met by the input, enhances the operabilityof the device and makes the user-device interface more efficient (e.g.,by providing improved feedback and reducing user mistakes whenoperating/interacting with the device) which, additionally reduces powerusage and improves battery life of the device by enabling the user touse the device more quickly and efficiently.

In accordance with method 8000 and with reference to FIG. 8B, inaccordance with a determination that the input meets the first criteria,the device selects (8014) the first tactile output profile for the firsttactile output in accordance with a current value of the firstparameter. Similarly, in some embodiments, in accordance with adetermination that the input meets the second criteria, the deviceselects the second tactile output profile for the second tactile outputin accordance with a current value of the first parameter. In someembodiments, the device continuously and dynamically adjusts thefirst/second tactile output profile when adjusting the first parameterin accordance with the input. In some embodiments, in accordance with adetermination that the input meets the first/second criteria, the deviceselects the audio output profiles for audio outputs that accompanies thefirst/second tactile outputs, in accordance with a current value of thefirst parameter. In some embodiments, the device continuously anddynamically adjusts the audio output profile when adjusting the firstparameter in accordance with the input. Providing localized tactileoutput in the input region with a tactile output profile thatcorresponds to a current value of the parameter that is being modifiedby the input enhances the operability of the device and makes theuser-device interface more efficient (e.g., by providing improvedfeedback and reducing user mistakes when operating/interacting with thedevice) which, additionally reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 8000 and with reference to FIG. 8B, inaccordance with a determination that the current value of the firstparameter has reached an end of a value range of the first parameter(e.g., volume value reaching maximum volume value in FIG. 5B4), thedevice generates (8016) an end tactile output (e.g., tactile output 526in FIG. 5B4) with a respective tactile output profile that is distinctfrom the first (and, optionally, the second) tactile output profile. Forexample, when the value of the first parameter reaches a maximum orminimum in a value range of the first parameter, the tactile output hasa quality that is discrete and distinct from the series of tactileoutputs that have been generated when the value of the first parameteris continuously adjusted in response to the input on the first/secondsub-regions of the input region. In some embodiments, the end tactileoutput has a different tactile output pattern from the first tactileoutput. In some embodiments, the end tactile output has a differentaccompanying audio output from the audio output accompanying the firsttactile output. In some embodiments, before reaching the end of thevalue range of the first parameter and generating the end tactileoutput, the device continuously adjusts a characteristic (e.g.,frequency, number of cycles, amplitude) of the first tactile outputprofile in accordance with a continuous change of the current value ofthe first parameter within the value range of the first parameter. Insome embodiments, no tactile output is generated when the current valueof the first parameter was already at an end of the value range for thefirst parameter before the first criteria are met by the input. Forexample, an end tactile output is generated when the device increases acurrent value of the first parameter to an end of the value range inresponse to a press input, but the device forgoes generating additionaltactile outputs when the user presses the input region one or moreadditional times, because no change in parameter value is made by theadditional press inputs. A localized tactile output with a distincttactile output profile is provided when an end value of the parameter isreached. The tactile output profile for the tactile output correspondingto the end value of the parameter is different from the tactile outputprofile used for tactile outputs generated in conjunction with arrivingat other values of the parameter prior to reaching the end value. Usinga distinct tactile output profile to indicate that the end value of theparameter has been reached enhances the operability of the device andmakes the user-device interface more efficient (e.g., by providingimproved feedback and reducing user mistakes when operating/interactingwith the device) which, additionally reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 8000 and with reference to FIG. 8B, inaccordance with a determination that the current value of the firstparameter has reached an end of a value range of the first parameter,the device forgoes (8018) generating the first (and, optionally, thesecond) tactile output profile. Providing differentiated feedback formodification of a device parameter in response to an input, includingproviding a tactile output in conjunction with reaching a new value thatis not an end value of the parameter, and forgoing provision of thetactile output when an end value of the parameter is reached in responseto the input enhances the operability of the device and makes theuser-device interface more efficient (e.g., by providing improvedfeedback and reducing user mistakes when operating/interacting with thedevice) which, additionally reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 8000 and with reference to FIG. 8B, inaccordance with a determination that the input did not increase abovethe first intensity threshold: in accordance with a determination thatthe input is detected in the first sub-region of the input region, thedevice generates (8020) first feedback (e.g., a first visual feedbackand/or a third tactile output with a third tactile output pattern)indicating that the input is detected in the first sub-region of theinput region; and in accordance with a determination that the input isdetected in the second sub-region of the input region, the devicegenerating second feedback that is distinct from the first feedback(e.g., a second visual feedback and/or a fourth tactile output with afourth tactile output pattern) indicating that the input is detected inthe second sub-region of the input region. In some embodiments, thefirst visual feedback includes highlighting a first region of a userinterface object (e.g., an upper portion of a slider control, or anupward arrow), and the second visual feedback includes highlighting asecond region of the user interface object (e.g., a lower portion of theslider control, or a downward arrow). In some embodiments, the third andfourth tactile outputs have the same tactile output profiles anddifferent accompanying audio outputs (e.g., one with a higher frequencythan the other). In some embodiments, the third tactile output and thefourth tactile outputs have distinct tactile output profiles (e.g.,frequencies, patterns, amplitudes, etc.). In some embodiments, the thirdtactile output and the fourth tactile output are generated by the samelocalized tactile output generator at the input region (e.g., the sidebutton region), and the tactile outputs changes depending on thelocation that the user is making contact with the input region. In someembodiments, separate localized tactile output generators are used togenerate localized tactile outputs at each of the first sub-region andthe second sub-region of the input region. Providing differentiatedfeedback to indicate which one of two sub-regions of the input region isbeing touched before the sub-region is activated by a press inputenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by providing improved feedback andreducing user mistakes when operating/interacting with the device)which, additionally reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 8000 and with reference to FIG. 8C, modifyingthe first parameter of the device in the first manner includes (8022)modifying the first parameter of the device in a first direction (e.g.,an upward direction, a direction with increasing values, a clockwisedirection, a leftward direction, etc.); and modifying the firstparameter of the device in the second manner includes modifying thefirst parameter of the device in a second direction that is distinctfrom (e.g., opposite of) the first direction (e.g., a downwarddirection, a direction with decreasing values, a counter-clockwisedirection, a rightward direction, etc.). This is illustrated in FIGS.5B1-5B10, where volume level is increased by press input detected on theupper input region 456, and volume level is decreased by press inputdetected on the lower input region 457. For example, for each pressinput (e.g., as defined by a duration of the contact or a rise-and fallof the intensity across an intensity threshold) detected on the upperregion of the input region, the device increases the first parameter(volume, brightness, or other control parameter of the device) by afixed amount; and for each press input detected on the lower region ofthe input region, the device decreases the first parameter by the fixedamount. In some embodiments, the amount that is changed for each pressinput is adjusted by maximum intensity of the press input (e.g., ahigher intensity corresponds to a larger multiplier for the change inthe first parameter of the device). Modifying the parameter in twodistinct directions depending on which one of two sub-regions of theinput region is being activated by a press input enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by reducing the number of inputs required to achieve anintended result) which, additionally reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 8000 and with reference to FIG. 8C, modifyingthe first parameter of the device in the first manner includes (8024)modifying the first parameter of the device by a first amount (e.g., afirst multiplier of the input, a first absolute amount, etc.); andmodifying the first parameter of the device in the second mannerincludes modifying the first parameter of the device by a second amountthat is distinct from (e.g., greater than, or smaller than, or apredefined multiple of) the first amount (e.g., a second multiplier ofthe input, a second absolute amount, etc.). For example, for each pressinput (e.g., as defined by a duration of the contact or a rise-and fallof the intensity across an intensity threshold) detected on a firstupper region of the input region, the device increases the firstparameter (volume, brightness, or other control parameter of the device)by a first fixed amount; and for each press input detected on a secondupper region of the input region below the first upper region, thedevice increases the first parameter by a second fixed amount (e.g., 5%at a time, or 5 times at a time, etc.) that is smaller than the firstfixed amount (e.g., 10% at a time, or 10 times at a time, etc.). In someembodiments, for each press input detected on a first lower region ofthe input region, the device decreases the first parameter (volume,brightness, or other control parameter of the device) by a third fixedamount; and for each press input detected on a second lower region ofthe input region below the first lower region, the device decreases thefirst parameter by a fourth fixed amount (e.g., 10% at a time, or 10times at a time, etc.) that is greater than the first fixed amount(e.g., 5% at a time, or 5 times at a time, etc.). In some embodiments,the amount that is changed for each press input is adjusted by maximumintensity of the press input (e.g., a higher intensity corresponds to alarger multiplier for the change in the first parameter of the device).Modifying the parameter by two distinct amounts depending on which oneof two sub-regions of the input region is being activated by a pressinput enhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by reducing the number of inputsrequired to achieve an intended result) which, additionally reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 8000 and with reference to FIG. 8C, modifyingthe first parameter of the device in the first manner includes (8026):dynamically adjusting a first rate by which the first parameter of thedevice is modified, in accordance with a characteristic intensity of theinput (e.g., an amount by which the intensity of the input exceeds thefirst intensity threshold) that is detected at the first sub-region ofthe input region; and modifying the first parameter of the device in thesecond manner includes: dynamically adjusting a second rate by which thefirst parameter of the device is modified, in accordance with acharacteristic intensity of the input (e.g., an amount by which theintensity of the input exceeds the first intensity threshold) that isdetected at the second sub-region of the input region. Dynamicallyadjusting a rate by which the parameter is adjusted in response to theinput in accordance with a characteristic intensity of the inputenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by reducing the number of inputsrequired to achieve an intended result) which, additionally reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 8000 and with reference to FIG. 8C, the firstparameter is (8028) a control parameter for operating a hardwarecomponent of the device (e.g., a volume for speaker (e.g., by operatingcontrol 520 in FIG. 5B7), a brightness for display (e.g., by operatingcontrol 550 in FIG. 5B11), or flashlight, etc.). Modifying a controlparameter for a hardware component of the device in different manners inaccordance with a determination of which one of two sub-regions of theinput region is activated by the press input enhances the operability ofthe device and makes the user-device interface more efficient (e.g., byreducing number of inputs required to achieve an intended result),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 8000 and with reference to FIG. 8C, the firstparameter is (8030) an operation control parameter for moving throughcontent (e.g., scrolling up and down in a content item, scanning througha listing of objects (e.g., images, media objects, etc.), paging up anddown paginated content, playback or scrubbing through a media object(e.g., a music clip, a video file, etc.)). Moving through content (e.g.,scrolling up and down, or scrubbing through media content or a list) indifferent manners in accordance with a determination of which one of twosub-regions of the input region is activated by the press input enhancesthe operability of the device and makes the user-device interface moreefficient (e.g., by reducing number of inputs required to achieve anintended result), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 8000 and with reference to FIG. 8C, inaccordance with a determination that the input meets the first criteriaa first number of times (e.g., the device detects that the intensity ofthe input rises and falls, and crosses the first intensity threshold thefirst intensity threshold multiple times), the device modifies (8032)the first parameter of the device in the first manner a second number oftimes (e.g., each time by the first amount, and for each increase abovethe first intensity threshold) that corresponds to the first number oftimes. For example, in FIGS. 5B2-5B4, volume levels are increasedmultiple times in response to multiple consecutive press input detectedon input region 456. Similarly, in accordance with a determination thatthe input meets the second criteria a third number of times (e.g., thedevice detects that the intensity of the input rises and falls, andcrosses the first intensity threshold the first intensity thresholdmultiple times), the device modifies the first parameter of the devicein the second manner a fourth number of times (e.g., each time by thefirst amount, and for each increase above the first intensity threshold)that corresponds to the third number of times. Modifying the parameterin a respective manner by a number of times that corresponds to thenumber of times that the input meets the first criteria enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 8000 and with reference to FIG. 8D, inresponse to detecting the input on the input region: in accordance witha determination that the input meets third criteria (e.g., firstenhanced button activation criteria), wherein the third criteria requirean increase in intensity of the input above a second intensity thresholdthat is above the first intensity threshold (e.g., a deep pressintensity threshold that is above the contact detection intensitythreshold and the light press intensity threshold) while the input isdetected in the first sub-region of the input region, in order for thethird criteria to be met, the device resets (8034) a first parameter ofthe device to a first predetermined value (e.g., maximize the volumelevel (e.g., as illustrated in FIGS. 5B8-5B9), or scroll to the top,without going through the intermediate values). In some embodiments, inresponse to detecting the input on the input region: in accordance witha determination that the input meets fourth criteria (e.g., secondenhanced button activation criteria), wherein the fourth criteriarequire an increase in intensity of the input above the second intensitythreshold while the input is detected in the second sub-region of theinput region, in order for the fourth criteria to be met, resetting thefirst parameter of the device to a second predetermined value that isdistinct from the first predetermined value. For example, whenscrollable content (e.g., scrollable document) or scrub-able content(e.g., music clip) is displayed, a deep press on the upper portion ofthe side button causes the device to skip to the beginning of content,while a deep press on the lower portion of the side button causes thedevice to skip to the end of the content. When a list of items aredisplayed, a deep press on the upper portion of the side button causesthe device to skips to the first item in the list, and a deep press onthe lower portion of the side button causes the device to skip to thelast item in the list. When a slider control is presented, a deep presson the upper portion of the side button causes the device to reset theindicator to the beginning of the slider control, and a deep press onthe lower portion of the side button causes the device to reset theindicator to the end of the slider control. When a filter is beingapplied, a deep press on the upper portion of the side button causes amaximum value of a range of filter values to be applied, and a deeppress on the lower portion of the side button causes a minimum value ofthe range of filter values to be applied. Resetting the parameter to apredetermined value when the input meets an enhanced intensityrequirement (e.g., as required by the third criteria) while forgoingadjusting the parameter in the usual manner (e.g., the manner ofadjustment made in response to a press input that does not meet theenhanced intensity requirement) enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by reducingthe number of inputs required to achieve an intended result), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 8000 and with reference to FIG. 8D, inresponse to detecting the input (e.g., the input by contact 518 on upperinput region 456 in FIG. 5B2) on the input region: the device displays(8036) a first user interface object (e.g., volume control 520 in FIG.5B2) that corresponds to the first parameter (e.g., device volume) thatis modifiable through a required input on the input region (e.g., theuser interface object is a scroll bar, a volume control, a brightnesscontrol, a scrubber, a selection indicator that can move through a listof items), wherein the user interface object is displayed withoutrequiring an intensity of the input to exceed the first intensitythreshold. In some embodiments, the first user interface object was notdisplayed before detecting the input. In some embodiments, the firstuser interface object is displayed in a first manner before the input isdetected on the input region, and displayed in a second manner (e.g.,highlighted or selected) after the input is detected on the inputregion. Displaying a user interface object that corresponds to theparameter that is to be modified in response to a touch input detectedon the input region without requiring an intensity of the touch input toexceed the intensity threshold for actually modifying the parameterenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by providing improved feedback andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 8000 and with reference to FIG. 8D, inresponse to detecting the input on the input region: in accordance witha determination that the input (e.g., input by contacts 542 and 544 inFIGS. 5B10-5B11) meets fifth criteria (e.g., parameter switchingcriteria), the device switches (8038) a current selected parameter thatis modifiable through a required input on the input region from thefirst parameter (e.g., volume control as shown in FIG. 5B10) to a secondparameter (e.g., brightness control as shown in FIG. 5B11) that isdistinct from the first parameter. For example, initially, the currentlyselected parameter that is modifiable by pressing on the input region isthe volume of the device. In response to detecting a press input on boththe upper and lower portions of the input region simultaneously, thecurrently selected parameter that is modifiable by pressing on the inputregion becomes the brightness of the display. In some embodiments, alisting of parameters are controllable through press inputs on the inputregion, and the device cycles through the list to select the parameterthat is currently modifiable each time that the device detects both theupper portion and the lower portion of the input region are pressedsimultaneously. In some embodiments, the device also displays the userinterface object that corresponds to the currently selected parameterthat is controlled by the input region. Switching the parameter that isto be modified in response to a touch input detected on the input regionthat meet preset criteria (e.g., parameter switching criteria) enhancesthe operability of the device and makes the user-device interface moreefficient (e.g., by providing additional control options withoutcluttering the user interface and reducing the number of inputs requiredto achieve an intended result), which, additionally, reduces power usageand improves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In accordance with method 8000 and with reference to FIG. 8D, the fifthcriteria require (8040) an increase in intensity of the input above thefirst intensity threshold (e.g., a light press intensity threshold thatis above the contact detection intensity threshold and below a deeppress intensity threshold) while the input is concurrently detected inthe first sub-region and the second sub-region of the input region, inorder for the fifth criteria to be met. This is illustrated in FIGS.5B10-5B11. In some embodiments, instead of requiring a press input onboth the upper and the lower portion of the input region simultaneously,the device switches the parameter that that input region controls inresponse to a swipe input on the input region (e.g., a downward swipe oran upward swipe along the longitudinal direction of the input region).Switching the parameter that is to be modified in response to a pressinput concurrently detected on both sub-regions of the input regionenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by providing additional control optionswithout cluttering the user interface, and by reducing user mistakeswhen operating/interacting with the device), which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIGS. 8A-8D have been described is merely exemplary and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 6000, 7000, 9000, 10000, 11000, and 12000) are also applicablein an analogous manner to method 8000 described above with respect toFIGS. 8A-8D. For example, the contacts, gestures, user interfaceobjects, tactile outputs, audio outputs, intensity thresholds, timethresholds, focus selectors, and animations described above withreference to method 8000 optionally have one or more of thecharacteristics of the contacts, gestures, user interface objects,tactile outputs, audio outputs, intensity thresholds, time thresholds,focus selectors, and animations described herein with reference to othermethods described herein (e.g., methods 6000, 7000, 9000, 10000, 11000,and 12000). For brevity, these details are not repeated here.

FIGS. 9A-9C are flow diagrams illustrating a method 9000 of guidingdevice movement in accordance with some embodiments.

Method 9000 relates providing localized tactile outputs using tactileoutput generators located at different locations of the device to guideand prompt movement of the device in order to continue performance of arespective task (e.g., completing a panorama picture, navigating to adestination, searching for an object), where the respective task isperformed during a media capturing mode of operation of the device, andthe tactile output generators are selectively activated to prompt themovement of the device in a respective direction in accordance with adirection by which one or more objects in the field of view of thecamera need to be shifted in order to continue performance of the task.Using localized tactile outputs to guide and prompt the user to move thedevice in a respective direction based on the direction by which anobject in the field of view of the camera needs to be shifted tocontinue performance of the task enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by providingimproved feedback to help the user to provide required inputs to achievean intended outcome, and by reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

The method 9000 is performed at an electronic device (e.g., device 300,FIG. 3, or portable multifunction device 100, FIGS. 1A and 4A) with adisplay, one or more cameras, and a plurality of tactile outputgenerators including a first tactile output generator at a firstlocation in the device (e.g., on or underneath a first edge of thedevice (e.g., right edge, or top edge)) and a second tactile outputgenerator at a second location in the device (e.g., on or underneath asecond edge of the device (e.g., left edge, or bottom edge)). In someembodiments, the first tactile output generator and the second tactileoutput generator are located in separate sections along the same edge ofthe device.

In accordance with method 9000 and with reference to FIG. 9A, while thedevice is in a media capture mode of operation (e.g., a panoramacapturing mode in FIG. 5E1, or video recording mode in FIG. 5F3) (e.g.,continuously capturing video or still images using the one or morecameras), the device performs (9002) a respective task (e.g., capturinga panorama image, providing turn by turn navigation instructions on avehicle, searching for a particular object in the surroundingenvironment (e.g., pairing smart home devices), capturing an image of anobject or person with predefined framing and alignment requirements(e.g., for biometric authentication, for certain artistic composition),providing vehicle steering guidance (e.g., reverse parking,releasing/pickup load on target, etc.), providing guidance for remotenavigation control for controlling unmanned vehicles (e.g., an aerialdrone), providing guidance for camera-assisted surgery, etc.).

While performing the respective task, the device detects (9006) that oneor more objects (e.g., predefined objects, visual markers, features,landmarks, etc.) in a field of view of the one or more cameras of thedevice meet first criteria (e.g., device-shifting criteria). Forexample, in FIGS. 5E2 and 5E3, the mountains in field of view deviatefrom horizontal movement. In FIG. 5E4, movement of the mountains in thefield of view is stalled or stopped before completion of the panoramaimage. In FIGS. 5F4 and 5F5, faces 505-3 and 505-3 are about to move outof frame. In some embodiments, the first criteria are used to determinewhether movement of the field of view of the cameras is required inorder to satisfy the requirements for performing the respective task.Depending on the respective task, the requirements may be different. Forexample, when capturing a panorama image, the requirement is for thefield of view to shift horizontally (e.g., parallel to the ground), andthe first criteria are met when the device detects a deviation from thehorizontal movement as determined based on changing positions of one ormore objects in the field of view of the camera. In another example,when providing turn-by-turn navigation instructions on a vehicle, one ofthe requirements is to have the field of view of the camera on a correctlane based on the next turn (e.g., right lane for an upcoming rightturn, and left lane for an upcoming left turn), and the first criteriaare met when the device detects that vehicle is not in the correct lane(e.g., based on the road markings identified in the field of view of thecameras). In another example, when capturing an image for a photo ID,the requirement is that a subject's head needs to fit within apredefined box in the image; and the first criteria are met when thedevice detects that the subjects head is not within the predefined box(e.g., off-center or too large) in the field of view of the camera.

In response to detecting that the one or more objects in the field ofview of the one or more cameras meet the first criteria (9006), thedevice generates, with one or more of the plurality of tactile outputgenerators, a tactile output (e.g., tactile output 523 in FIG. 5E2,tactile output 524 in FIG. 5E3, tactile output 519 in FIG. 5E4, tactileoutput 601 in FIG. 5F4, tactile output 603 in FIG. 5F5) promptingmovement of the field of view of the one or more cameras in a respectivedirection (e.g., through moving (e.g., translation in 2D, translation in3D, rotation, panning, pivoting, tilting, or combinations of the above,etc.) the device as a whole or a part of the device that includes thecamera(s)), wherein generating the tactile output prompting movement ofthe field of view of the one or more cameras in the respective directionincludes: in accordance with a determination, based on the one or moreobjects in the field of view of the camera, that the field of view ofthe camera needs to be shifted in a first direction (e.g., leftward,clockwise, upward, forward, in the northwest direction, closer to user,or a combination of two or more of the above, etc.) to continueperforming the respective task (e.g., to continue capturing of thepanorama image, to continue providing turn by turn navigationinstructions on the vehicle, to continue searching for the particularobject in the surrounding environment (e.g., pairing smart homedevices), to continue capturing of the image of the object or personwith the predefined framing and alignment requirements (e.g., forbiometric authentication, for certain artistic composition), to continueproviding vehicle steering guidance (e.g., reverse parking,releasing/pickup load on target, etc.), to continue providing guidancefor remote navigation control for controlling unmanned vehicles (e.g.,an aerial drone), to continue providing guidance for camera-assistedsurgery, etc.), generating a first tactile output (e.g., the firsttactile output is provided by the first tactile output generator, and islocalized at the first location of in the device (e.g., a left edge ofthe device)) that prompts movement of the field of view of the one ormore cameras in the first direction (e.g., tactile output 523 in FIG.5E2 upward prompts movement of the field of view; or tactile output 601in FIG. 5F4 prompts leftward movement of the field of view); and inaccordance with a determination, based on the one or more objects in thefield of view of the camera, that the field of view of the camera needsto be shifted in a second direction (e.g., rightward, counter-clockwise,downward, backward, in the southeast direction, farther away from theuser, or a combination of two or more of the above, etc.), differentfrom the first direction, to continue performing the respective task,generating a second tactile output (e.g., the second tactile output isprovided by the second tactile output generator, and is localized at thesecond location of in the device (e.g., a right edge of the device)),different from the first tactile output, that prompts movement of thefield of view of the one or more cameras in the second direction (e.g.,tactile output 521 in FIG. 5E3 prompts downward movement of the field ofview and tactile output 519 in FIG. 5E4 prompts forward movement of thefield of view; or tactile output 603 in FIG. 5F5 prompts rightwardmovement of the field of view).

In accordance with method 9000 and with reference to FIG. 9A, generatingthe first tactile output includes (9008) generating the first tactileoutput using the first tactile output generator at the first location inthe device (e.g., on or underneath a first edge of the device (e.g.,right edge, or top edge)); and generating the second tactile outputincludes generating the second tactile output generator using the secondtactile output generator at the second location in the device (e.g., onor underneath a second edge of the device (e.g., left edge, or bottomedge)). In some embodiments, the first tactile output generator and thesecond tactile output generator are located in separate sections alongthe same edge of the device. In some embodiments, for a respective task,each of a plurality of directions relevant to the task are associatedwith a respective subset (e.g., one or more) of the plurality of tactileoutput generators based on the respective locations of the tactileoutput generators in the device; and the device selectively activates asubset of the tactile output generators in accordance with the directionthat the field of view of the camera(s) need to be moved to continueperforming the respective task. In some embodiments, the first andsecond tactile outputs have the same tactile output profiles (e.g.,frequency, amplitude, pattern, accompanying audios, etc.), but providedifferent prompt to the user because they are generated by differenttactile output generators located at different locations in the deviceand are localized at different portions on the surface of the device.Using tactile output generators located at different locations on thedevice to produce respective localized tactile outputs to prompt theuser to move the device in difference directions enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide required inputs toachieve an intended outcome, and by reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 9000 and with reference to FIG. 9B, generatingthe first tactile output (e.g., for prompting clockwise movement)includes (9010) generating the first tactile output using the firsttactile output generator and the second tactile output generator inaccordance with a first output sequence (e.g., the first sequenceincludes the first tactile output generator followed by the secondtactile output generator). Generating the second tactile output (e.g.,for prompting counterclockwise movement) includes generating the secondtactile output using the first tactile output generator and the secondtactile output generator in accordance with a second output sequencethat is distinct from the first output sequence (e.g., the secondsequence includes the second tactile output generator followed by thefirst tactile output generator). This is illustrated in FIGS. 5E2, 5E3,and 5E4, where coordinated tactile outputs 523, 521, and 519 are eachgenerated using a pair of tactile output generators located at differentinput regions in accordance with different output sequences, forexample. This is further illustrated in FIGS. 5F4 and 5F5, wherecoordinated tactile outputs 601 and 603 are each generated using a pairof tactile output generators located at different input regions inaccordance with different output sequences. In some embodiments, thefirst tactile output generator and the second tactile output generatorare located in separate sections along the same edge of the device, oron different edges of the device. In some embodiments, for a respectivetask, each of a plurality of directions relevant to the task areassociated with a respective operation sequence for all or a selectedsubset of the plurality of tactile output generators based on therespective locations of the tactile output generators in the device; andthe device activates all or the selected subset of the tactile outputgenerators in accordance with the operation sequence that corresponds tothe direction that the field of view of the camera(s) need to be movedto continue performing the respective task. In some embodiments, thefirst sequence includes a respective tactile output generator only once(e.g., for movement in the first direction, the sequence is one outputby the first tactile output generator followed by one output by thesecond tactile output generator, and then optionally repeat the firstsequence one or more times with a predetermined delay between eachrepeat to separate the sequences; and for movement in the seconddirection, the second sequence includes one output by the second tactileoutput generator followed by one output by the first tactile outputgenerator, and then optionally repeat the second sequence one or moretimes with a predetermined delay between each repeat to separate thesequences). In some embodiments, the sequence may call for a respectivetactile output generator multiple times (e.g., for the movement in thefirst direction, the first sequence includes two consecutive outputs bythe first tactile output generator followed by one output by the secondtactile output generator, and then optionally repeat the first sequenceone or more times with a predetermined delay between each repeat toseparate sequences; and for the movement in the second direction, thesecond sequence includes two consecutive outputs by the second tactileoutput generator followed by one output by the first tactile outputgenerator, then optionally repeat the second sequence one or more timeswith a predetermined delay between each repeat to separate thesequences. Operating a set of tactile output generators located atdifferent locations on the device in accordance with different outputsequences to prompt the user to move the device in difference directionsenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to provide requiredinputs to achieve an intended outcome, and by reducing user mistakeswhen operating/interacting with the device), which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In accordance with method 9000 and with reference to FIG. 9B, generatingthe first tactile output (e.g., for prompting clockwise movement)includes (9012) generating the first tactile output using the firsttactile output generator and the second tactile output generator inaccordance with a first power ratio (e.g., the first power ratioincludes the first tactile output generator operating with 20% power ofthe second tactile output generator). Generating the second tactileoutput (e.g., for prompting counterclockwise movement) includesgenerating the second tactile output using the first tactile outputgenerator and the second tactile output generator in accordance with asecond power ratio that is distinct from the first power ratio (e.g.,the second power ratio includes the second tactile output generatoroperating with 20% of the first tactile output generator). This isillustrated in FIGS. 5E2 and 5E3, where coordinated tactile outputs 523,521, and 519 are each generated using a pair of tactile outputgenerators located at different input regions in accordance withdifferent power ratios, for example. In some embodiments, the firsttactile output generator and the second tactile output generator arelocated in separate sections along the same edge of the device, or ondifferent edges of the device. In some embodiments, for a respectivetask, each of a plurality of directions relevant to the task areassociated with a respective power ratio for all or a selected subset ofthe plurality of tactile output generators based on the respectivelocations of the tactile output generators in the device; and the deviceactivates all or the selected subset of the tactile output generators inaccordance with the power ratio that corresponds to the direction thatthe field of view of the camera(s) need to be moved to continueperforming the respective task. Operating a set of tactile outputgenerators located at different locations on the device in accordancewith different output power ratios to prompt the user to move the devicein difference directions enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser to provide required inputs to achieve an intended outcome, and byreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 9000 and with reference to FIG. 9B, detectingthat one or more objects in the field of view of the one or more camerasof the device meet first criteria includes (9014): identifying a firstobject (e.g., faces 505-1, 505-2, and 505-3 in FIG. 5F3) (e.g., asubject's head, a predefined road marking, a target, etc.) in the fieldof view of the one or more cameras of the device in accordance withpredefined object identification criteria (e.g., based on imageprocessing, feature extraction, and object recognition techniques); anddetecting that a current position or movement of the first object in thefield of view of the one or more cameras deviates from a predefinedposition or movement path (e.g., a center of the field of view, apredefined navigation path projected on the field of view) in the fieldof view of the one or more cameras (e.g., face 505-3 is within athreshold distance of a left edge of the field of view in FIG. 5F4 andface 505-2 is within a threshold distance of a right edge of the fieldof view in FIG. 5F5). For example, when performing the task of taking aphoto ID, the device detects that the first criteria are met by theuser's head in the field of view of the cameras when detecting that theuser's head is off-center in the field of view. In another example, whenproviding indoor navigation guidance, the device detects that the firstcriteria are met by an entrance in the field of view of the cameras whenthe device detects that the entrance is offset from a central region ofthe field of view by more than a threshold distance. In yet anotherexample, when providing obstacle avoidance guidance, the device detectsthat the first criteria are met by a moving obstacle when the devicedetects that the movement path of the moving obstacle has deviated froma predicted movement path, requiring an adjustment of the existingobstacle avoidance maneuver. In yet another example, when capturing apanorama image, the device detects that the first criteria are met bythe scene currently in the field of view of the camera when the devicedetects that the positions of previously captured portion of the sceneis shifted up or down relative to their previously imaged positions.Using deviation of an object in the field of view of the camera from apredefined position or movement path as the basis for prompting the userto move the device in a respective direction enhances the operability ofthe device and makes the user-device interface more efficient (e.g., byhelping the user to provide required inputs to achieve an intendedoutcome, and by reducing user mistakes when operating/interacting withthe device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 9000 and with reference to FIG. 9B, detectingthat one or more objects in the field of view of the one or more camerasof the device meet first criteria includes (9016): identifying a firstobject (e.g., a tracked moving object, such as a bird, a shooting start,a moving target, etc., or a stationary object, such as a fixed target, asubject of a portrait, etc.) in the field of view of the one or morecameras of the device in accordance with predefined objectidentification criteria (e.g., based on image processing, featureextraction, and object recognition techniques); and detecting that atleast a portion of the first object is outside of the field of view ofthe one or more cameras. In some embodiments, when tracking a movingtarget, the device detects that the moving object in the field of viewof the cameras met the first criteria when the device detects that themoving object is moving out of the field of view of the camera.Prompting the user to move the device in a respective direction inaccordance with a determination that at least a portion of a predefinedobject in the field of view of the camera has moved out of the field ofview of the camera enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide required inputs to achieve an intended outcome, and by reducinguser mistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 9000 and with reference to FIG. 9C, detectingthat one or more objects in the field of view of the one or more camerasof the device meet first criteria includes (9018): identifying a firstobject (e.g., an object in a partial scene in a panorama, etc.) in thefield of view of the one or more cameras of the device in accordancewith predefined object identification criteria (e.g., based oncomparison between the current field of view and previously capturedportions of the panorama image); and detecting that movement speed ofthe first object is outside of a predefined speed range (e.g., as shownin FIG. 5E4 where the mountain in the field of view is moving too slowlyrelative to the device (e.g., because the device movement is stalled))(e.g., too fast or too slow to take a clear panorama image). In someembodiments, when tracking a moving target, the device detects that themoving object in the field of view of the cameras met the first criteriawhen the device detects that the moving object is moving faster orslower that the device by a threshold speed difference. Prompting theuser to move the device in a respective direction in accordance with adetermination that a predefined object in the field of view of thecamera is moving with a speed that is outside of a predefined speedrange enhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to provide requiredinputs to achieve an intended outcome, and by reducing user mistakeswhen operating/interacting with the device), which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In accordance with method 9000 and with reference to FIG. 9C, the devicedetects (9020) that the first criteria are met while the device isstationary (e.g., the object(s) moves in the field of view as a resultof motion by objects alone (e.g., when the device remains stationary inthe physical environment)). Prompting the user to move the device inaccordance with a determination that a predefined object in the field ofview of the camera meets the first criteria while the device isstationary enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide required inputs to achieve an intended outcome, and by reducinguser mistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 9000 and with reference to FIG. 9C, the devicedetects (9022) that the first criteria are met while the device is inmotion (e.g., the object(s) moves in the field of view as a result ofmotion by the device alone (e.g., when the object(s) remain stationaryin the physical environment)). Prompting the user to move the device inaccordance with a determination that a predefined object in the field ofview of the camera meets the first criteria while the device is inmotion enhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to provide requiredinputs to achieve an intended outcome, and by reducing user mistakeswhen operating/interacting with the device), which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In accordance with method 9000 and with reference to FIG. 9C, the devicedetects (9024) that the first criteria are met while the device and theobject are both in motion (e.g., the object(s) moves in the field ofview as a result of relative motion between the device and the object(s)(e.g., due to their different speeds)). Prompting the user to move thedevice in accordance with a determination that a predefined object inthe field of view of the camera meets the first criteria while both thedevice and the object are in motion enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide required inputs to achieve an intendedoutcome, and by reducing user mistakes when operating/interacting withthe device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 9000 and with reference to FIG. 9C, inresponse to detecting that the one or more objects in the field of viewof the one or more cameras meet the first criteria, the devicedisplaying a visual indicator prompting movement of the field of view ofthe one or more cameras in the respective direction (e.g., throughmoving (e.g., translation in 2D, translation in 3D, rotation, panning,pivoting, tilting, or combinations of the above, etc.) the device as awhole or a part of the device that includes the camera(s)), whereingenerating the visual indicator prompting movement of the field of viewof the one or more cameras in the respective direction includes (9026):in accordance with a determination, based on the one or more objects inthe field of view of the camera, that the field of view of the cameraneeds to be shifted in the first direction (e.g., leftward, clockwise,upward, forward, in the northwest direction, closer to user, or acombination of two or more of the above, etc.) to continue performingthe respective task, providing the visual indicator with a first visualcharacteristic (e.g., a first motion pattern, direction, shape,animation sequence, etc.) that prompts movement of the field of view ofthe one or more cameras in the first direction; and in accordance with adetermination, based on the one or more objects in the field of view ofthe camera, that the field of view of the camera needs to be shifted inthe second direction (e.g., rightward, counter-clockwise, downward,backward, in the southeast direction, farther away from the user, or acombination of two or more of the above, etc.) to continue performingthe respective task, generating the visual indicator with a secondvisual characteristic (e.g., a second motion pattern, direction, shape,animation sequence, etc.), different from the first visualcharacteristic, that prompts movement of the field of view of the one ormore cameras in the second direction. Providing visual feedback inconjunction with the tactile output prompting the user to move thedevice in a respective direction, and providing the visual feedback witha visual characteristic corresponding to the respective directionenhance the operability of the device and make the user-device interfacemore efficient (e.g., by helping the user to provide required inputs toachieve an intended outcome, and by reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIGS. 9A-9C have been described is merely exemplary and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 6000, 7000, 8000, 10000, 11000, and 12000) are also applicablein an analogous manner to method 9000 described above with respect toFIGS. 9A-9C. For example, the contacts, gestures, user interfaceobjects, tactile outputs, audio outputs, intensity thresholds, timethresholds, focus selectors, and animations described above withreference to method 9000 optionally have one or more of thecharacteristics of the contacts, gestures, user interface objects,tactile outputs, audio outputs, intensity thresholds, time thresholds,focus selectors, and animations described herein with reference to othermethods described herein (e.g., methods 6000, 7000, 8000, 10000, 11000,and 12000). For brevity, these details are not repeated here.

FIGS. 10A-10C are flow diagrams illustrating a method 10000 ofgenerating warning feedback for an impending operation in accordancewith some embodiments.

Method 10000 relates to generating a warning tactile output when firstcriteria are met by an input on an input region but second criteria arenot met by the input, and performing an operation when the secondcriteria (e.g., with an enhanced requirement relative to the firstcriteria) are met by the input. Providing the warning tactile outputusing a tactile output generator associated with the input region beforethe second criteria are met enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser to provide required inputs to achieve an intended outcome, andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently. In addition, forgoing generating the tactile output whenthe first criteria are not met, and forgoing performing the operationwhen a cancelation input is detected before the second criteria are metenhance the operability of the device and make the user-device interfacemore efficient (e.g., by reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

The method 10000 is performed at an electronic device (e.g., device 300,FIG. 3, or portable multifunction device 100, FIGS. 1A and 4A) with adisplay, and an input region that is associated with a tactile outputgenerator (e.g., a localized tactile output generator that is located atan edge of the device (e.g., a side button region) and that generateslocalized tactile outputs as opposed to whole-device tactile outputs).In some embodiments, the localized tactile output generator is a surfaceoscillator that rocks back and forth around a fixed pivot underneath theinput region or other tactile output generator that is capable ofgenerating tactile outputs that are directed specifically toward theinput region. In some embodiments, the localized tactile outputgenerator provides localized vibrations at the input region but not thedevice as a whole. The localized vibrations have varying amplitudes atdifferent locations on the device with greater amplitudes on the inputregion where a user's finger will typically rest while operating theinput region and with lower amplitudes at other places outside of theinput region where a user's hand will not typically rest while operatingthe input region. In some embodiments, the device further includes awhole-device tactile output generator (e.g., a mass located within thedevice and is moved back and forth around a central rest location) thatgenerates whole-device tactile outputs. In some embodiments, thewhole-device tactile output generator creates tactile outputs in manydifferent locations on the device at the same time, such as a tactileoutput generator that is coupled to a housing of the device. In someembodiments, the localized tactile output generator provides tactileoutputs that correspond to operations that are performed and/or visualfeedback that is generated in response to touch/press inputs detected onthe input region. In some embodiments, in conjunction with orindependent of the localized tactile outputs, the device furtherutilizes the whole-device tactile output generator to generate wholedevice tactile outputs that correspond to operations that are performedand/or visual feedback that is generated in response to touch/pressinputs detected on the touch-screen user interface, as opposed totouch/press inputs detected on the input region.

In accordance with method 10000 and with reference to FIG. 10A, thedevice detects (10002) an input on the input region (e.g., a touch-inputby a contact on the touch-screen display or the touch-sensitive sidebutton).

In accordance with method 10000 and with reference to FIG. 10A, inresponse to detecting the input on the input region: in accordance witha determination that first criteria (e.g., warning-initiation criteria)are met by the input on the input region, wherein the first criteriainclude a requirement that is met when a first input metric (e.g., atime characteristic of the input, an intensity characteristic of theinput, a motion characteristic of the input, and/or a hybridtime/intensity/motion characteristic of the input) exceeds a firstthreshold in order for the first criteria to be met (e.g., thewarning-initiation criteria are met by the input when the input meetsthe criteria for initiating performance of an operation, or the input isabout to meet the criteria for initiating performance of an operation ifthe input continues in its current course), the device generates (10006)a first tactile output with the tactile output generator associated withthe input region (e.g., the first tactile output is an escalatingtactile output with an increasing value for at least one of thecharacteristic parameters (e.g., amplitude, frequency, accompanyingaudio volume, accompanying audio frequency, etc.) of its tactile outputprofile). This is illustrated in FIG. 5G2, where an intensity of aninput by contact 605 has exceeded a light press intensity threshold; andthe device generates tactile output 607 to indicate that emergencycalling function will be triggered with continued increase in intensityof the input. Alternatively, this is illustrated in FIG. 5G3, where anintensity of an input by contact 605 has exceeded a deep press intensitythreshold and a negligible duration requirement, and the device starts acountdown period, and generates tactile output 609 (including inputregion 455) to indicate that emergency calling function will betriggered at the end of the countdown period. This is also illustratedin FIG. 5H3, where an input by contact 627 has met a light pressintensity threshold and a time threshold T; and the device generatestactile output 631 to indicate that digital assistant will soon becomeavailable to respond to user input. After generating the first tactileoutput: in accordance with a determination that the input meets secondcriteria (e.g., operation-execution criteria), wherein the secondcriteria include a requirement that is met when the first input metric(e.g., a time characteristic of the input, an intensity characteristicof the input, a motion characteristic of the input, and/or a hybridtime/intensity/motion characteristic of the input) exceeds a secondthreshold that is greater than the first threshold in order for thesecond criteria to be met, the device performs an operation (e.g.,executing a payment operation, making an emergency call, activating thevoice-based digital assistant, shut-down the device, etc.); and inaccordance with a determination that a cancelation input (e.g., lift-offof the contact from the input region, movement of the contact on theinput region, selection of a cancel button on the displayed userinterface, etc.) has been detected prior to the input meeting the secondcriteria, the device forgoes performance of the operation. This isillustrated in FIG. 5G3, where the intensity of the input by contact 605has exceeded a deep press intensity threshold greater than the lightpress intensity threshold; and the device generates tactile output 609to indicate that a countdown period for making an emergency call hasbeen started. If the input had been terminated before reaching the deeppress intensity threshold, the device treats the lift-off of contact 605as a cancellation input, and does not start the countdown period.Alternatively, this is illustrated in FIG. 5G4, where the intensity ofthe input by contact 605 is maintained above the deep press intensitythreshold for more than a threshold amount of time T (e.g., thecountdown period), and the device completes the countdown period andmakes the emergency call. If the input had been terminated or if theintensity of the input had dropped below the deep press intensitythreshold before the end of the countdown period, the device will notmake the emergency call. This is also illustrated in FIG. 5H5, where theintensity of the input by contact 627 is maintained above the lightpress intensity threshold for an additional time period required for thedigital assistant to be ready, the device presents the fully functionaldigital assistant when it is fully ready. If a cancellation input isdetected before the digital assistant is fully ready, the device doesnot present the digital assistant user interface as shown in FIG. 5H5.In response to detecting the input on the input region and in accordancewith a determination that the first criteria (e.g., warning-initiationcriteria) are not met by the input on the input region, the deviceforgoes (10008) generating the first tactile output.

In accordance with method 10000 and with reference to FIG. 10A, aftergenerating the first tactile output: in accordance with a determinationthat the input meets the second criteria (e.g., operation-executioncriteria), the device generates (10010) a second tactile output (e.g.,tactile output 609 to indicate start of the countdown in FIG. 5G3, ortactile output 615 to indicate making the emergency call in FIG. 5G4, ortactile output 641 to indicate that the digital assistant is ready inFIG. 5H5) indicating performance of the operation (e.g., entering aphase of the operation performance that is no longer cancellable). Forexample, a whole-device tactile output is generated by the internaltactile output generator that is distinct from the tactile outputgenerated with the input region to indicate execution of the paymentoperation, dialing the emergency call, activation of the voice-baseddigital assistant, starting the shut-down of the device, at which point,the operation is no longer cancellable. In some embodiments, the devicedoes not require lift-off of the contact from the input region in orderfor the second criteria to be met. In some embodiments, the first andsecond criteria are based on metrics that are distinct from a movementdistance of the contact (e.g., the first and second criteria are not metby a swipe input crossing different distance thresholds on the display,but are met by the inputs meeting intensity-based or time-based criteriainstead.). Generating a tactile output in conjunction with satisfactionof the second criteria which leads to performance of the operationenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by providing improved feedback andhelping the user to provide required inputs to achieve an intendedoutcome), which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 10000 and with reference to FIG. 10A, aftergenerating the first tactile output: in accordance with a determinationthat a cancellation input has been detected prior to the input meetingthe second criteria, the device ceases (10012) to generate the firsttactile outputs. For example, while the input is continuously detectedon the input region and/or continues to meet the first criteria, and/orbefore the cancellation input is detected, the device continues togenerate the first tactile output. The device ceases to generate thefirst tactile output when the second criteria are met or when thecancellation input is detected. Ceasing to generate the first tactileoutputs (e.g., the warning tactile outputs) when a cancellation inputhas been detected before the second criteria are met (e.g., before theperformance of the action is initiated and irreversible) enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by providing improved feedback and helping the user toprovide required inputs to achieve an intended outcome), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 10000 and with reference to FIG. 10B,generating the first tactile output includes (10014) generating acontinuous tactile output that is sustained (e.g., tactile output 609 inFIG. 5G3, or tactile output 631 in FIGS. 5H3 and 5H4) until the secondcriteria are met or when the cancelation input is detected. In someembodiments, a press input that does not meet the first criteria causesthe tactile output generator at the input region to generate abutton-press tactile output (e.g., a weak down-click tactile output)that is discrete and non-continuous, and a press input that meets thefirst criteria causes the tactile output generator at the input regionto generate a warning tactile output (e.g., a strong, sustained warningtactile output) that is continuous and the device forgoes generating thediscrete and noncontiguous tactile output. Generating a continuoustactile output (e.g., the warning tactile outputs) that is sustaineduntil the second criteria are met (e.g., when performance of theoperation is initiated and irreversible) or when a cancellation inputhas been detected before the second criteria are met (e.g., before theperformance of the action is initiated and irreversible) enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by providing improved feedback and helping the user toprovide required inputs to achieve an intended outcome), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 10000 and with reference to FIG. 10B,generating the continuous tactile output includes (10016) monotonicallychanging (e.g., monotonically increasing, or monotonically decreasing)at least one aspect of a tactile output profile corresponding to thefirst tactile output (e.g., increasing amplitude, increasing peakamplitude, increasing frequency, increasing volume for accompanyingaudio, increasing frequency for accompanying audio, etc.). For example,when the criteria for triggering warning for initiating an emergencycall are met by an input (e.g., a press input or a touch-hold input) onthe side button, the tactile output generator located at the side buttongenerates a continuous tactile output that has a sequence of vibrationswith increasing amplitudes and constant frequency, along with anaccompanying audio that has increasing frequency and increasing volume.In another example, when the criteria for triggering warning foractivating the voice-based digital assistant are met by an input on theside button, the tactile output generator located at the side buttongenerates a tactile output with a sequence of vibrations with anincreasing amplitude and a decreasing frequency, until the secondcriteria are met or when a cancelation input is detected. Generating acontinuous and monotonically changing (e.g., escalating) tactile output(e.g., the warning tactile outputs) that is sustained until the secondcriteria are met (e.g., when performance of the operation is initiatedand irreversible) or when a cancellation input has been detected beforethe second criteria are met (e.g., before the performance of the actionis initiated and irreversible) enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by providingimproved feedback and helping the user to provide required inputs toachieve an intended outcome), which, additionally, reduces power usageand improves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In accordance with method 10000 and with reference to FIG. 10B, thefirst criteria include (10018) a requirement that is met when acharacteristic intensity of the input exceeds a first intensitythreshold in order for the first criteria to be met (e.g., thewarning-initiation criteria are met by a light press input by a contacton the side button or on an affordance on the touch-sensitive display).In some embodiments, the first criteria further include a requirementthat the contact on the input region is kept substantially stationary(e.g., has less than a threshold amount of movement within a thresholdamount of time) during the time when the contact is initially detectedon the input region until the first criteria are met. Generating awarning tactile output when an intensity-based requirement of the firstcriteria are met by an input on an input region but second criteria arenot met by the input enhances the operability of the device and makesthe user-device interface more efficient (e.g., by helping the user toprovide required inputs to achieve an intended outcome, and reducinguser mistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 10000 and with reference to FIG. 10B, thesecond criteria include (10020) a requirement that is met when thecharacteristic intensity of the input exceeds a second intensitythreshold that is greater than the first intensity threshold in orderfor the second criteria to be met. For example, the input is provided bya contact with the input region, the device further requires that theinput includes less than a threshold amount of movement of the contact(e.g., same contact) for more than a threshold amount of time (e.g., thecontact is substantially stationary when the input is evaluated againstthe first and second criteria). Generating a warning tactile output whenan intensity-based requirement of the first criteria are met by an inputon an input region but an intensity-based second criteria (e.g., with anenhanced intensity requirement relative to the first criteria) are notmet by the input enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide required inputs to achieve an intended outcome, and reducinguser mistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 10000 and with reference to FIG. 10B, thefirst criteria include (10022) a requirement that is met when the inputis maintained for at least a first threshold amount of time with lessthan a threshold amount of movement in order for the first criteria tobe met (e.g., the warning-initiation criteria are met by a touch-holdinput (e.g., either with a concurrent intensity requirement or without aconcurrent intensity requirement) by a contact on the side button or onan affordance on the touch-sensitive display). Generating a warningtactile output when a time-based requirement (e.g., a touch-holdrequirement) of the first criteria are met by an input on an inputregion but second criteria are not met by the input enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide required inputs toachieve an intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 10000 and with reference to FIG. 10B, thesecond criteria include (10024) a requirement that is met when the inputis maintained for at least a second threshold amount of time that islonger than the first threshold amount of time with less than thethreshold amount of movement in order for the second criteria to be met.For example, the operation-execution criteria are met by a longtouch-hold input (e.g., either with a concurrent intensity requirementor without a concurrent intensity requirement) by a contact on the sidebutton or on an affordance on the touch-sensitive display). Generating awarning tactile output when a time-based requirement (e.g., a touch-holdrequirement) of the first criteria are met by an input on an inputregion but a time-based second criteria (e.g., with an enhanced timerequirement relative to the first criteria) are not met by the inputenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to provide requiredinputs to achieve an intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 10000 and with reference to FIG. 10C, thetactile output generator that is associated with the input regionresides (10026) at the input region and produces localized tactileoutputs at the input region. The device further includes a secondtactile output generator that does not reside at the input region andproduces whole-device tactile outputs (e.g., tactile outputs thatvibrates the entire device and are not localized at the whole-devicetactile output generator). The method 10000 includes: in response todetecting the input on the input region: in accordance with adetermination that the first criteria are met by the input on the inputregion, generating a third tactile output by the second tactile outputgenerator that is not localized at the input region (e.g., awhole-device tactile output that is detectable anywhere on the device),in addition to the first tactile output that is localized at the inputregion. Generating tactile outputs using both a localized tactile outputgenerator and a whole-device tactile output generator when firstcriteria are met by an input on an input region but second criteria(e.g., with an enhanced requirement relative to the first criteria) arenot met by the input enhances the operability of the device and makesthe user-device interface more efficient (e.g., by providing improvedfeedback to help the user provide required inputs to achieve an intendedoutcome, and reducing user mistakes when operating/interacting with thedevice), which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 10000 and with reference to FIG. 10C, thefirst tactile output is (10028) a discrete tactile output that simulatesactuation of the input region by the input; and wherein the thirdtactile output is a continuous tactile output that provides sustainedwarning feedback until the second criteria are met by the input or thecancelation input is detected. For example, the first tactile outputsimulates a button click when the first criteria are met by a pressinput on the side button, and the third tactile output is a vibrationthat is sustained and extends to the entire device (e.g., can bedetected by the user's hand holding the device at locations outside ofthe side button region). For example, when the operation is making anemergency call, after the user has pressed on the side button with therequired intensity (e.g., at least the light press intensity) for arequired period of time (e.g., 5 seconds), the warning-initiationcriteria are met by the input on the side button and the localizedtactile output generator at the side button generates a button-clicktactile output (e.g., on initial press, a light click tactile output isgenerated at the side button, and upon satisfaction of thewarning-initiation criteria, a strong click tactile output is generatedat the side button). In addition to the button-click tactile outputsgenerated by the localized tactile output generated at the side button,a sustained whole-device warning tactile output (e.g., sustained, strongvibrations) with increasing amplitude is generated by the internaltactile output generator, accompanied by a siren sound effect.Generating both a localized tactile output to simulate actuation of theinput region by the input and a sustained whole-device tactile output asa warning for an impending operation when first criteria are met by aninput on an input region but second criteria (e.g., with an enhancedrequirement relative to the first criteria) are not met by the inputenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by providing improved feedback to helpthe user provide required inputs to achieve an intended outcome, andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 10000 and with reference to FIG. 10C, inresponse to detecting the input on the input region: in accordance witha determination that first criteria (e.g., warning-initiation criteria)are met by the input on the input region, the device displays (10030) anindication (e.g., an animation, a count-down timer, a burning fuse, aprogress or intensity bar with a threshold indicator for triggeringperformance of the operation, etc.) that the operation will be performedwhen the input on the input region meets additional criteria (e.g., withincreased intensity, or be maintained for a longer time). For example,when the operation is activating the voice-based digital assistant,after the user has pressed on the side button with the requiredintensity (e.g., at least the light press intensity) for a requiredperiod of time (e.g., 1.5 seconds), the warning-initiation criteria aremet by the input on the side button and the localized tactile outputgenerator at the side button generates a long press button-click tactileoutput (e.g., on initial press, a light click tactile output isgenerated at the side button, and upon satisfaction of thewarning-initiation criteria, a strong click tactile output is generatedat the side button). In addition to the long press button-click tactileoutputs generated by the localized tactile output generated at the sidebutton, a sound and an onscreen animation are generated indicating thatthe digital assistant will become available with additional criteriabeing met by the input (e.g., an intensity meter that dynamicallyupdates to reflect the current contact intensity relative to theintensity threshold required to activate the digital assistant).Displaying an indication that the operation will be performed when theinput on the input region meets additional criteria after the firstcriteria are met by the input enhances the operability of the device andmakes the user-device interface more efficient (e.g., by providingimproved feedback to help the user provide required inputs to achieve anintended outcome, and reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

It should be understood that the particular order in which theoperations in FIGS. 10A-10C have been described is merely exemplary andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 6000, 7000, 8000, 9000, 11000, and 12000) are also applicable inan analogous manner to method 10000 described above with respect toFIGS. 10A-10C. For example, the contacts, gestures, user interfaceobjects, tactile outputs, audio outputs, intensity thresholds, timethresholds, focus selectors, and animations described above withreference to method 10000 optionally have one or more of thecharacteristics of the contacts, gestures, user interface objects,tactile outputs, audio outputs, intensity thresholds, time thresholds,focus selectors, and animations described herein with reference to othermethods described herein (e.g., methods 6000, 7000, 8000, 9000, 11000,and 12000). For brevity, these details are not repeated here.

FIGS. 11A-11B are flow diagrams illustrating a method 11000 of adjustingtactile outputs for an input on an input region depending on whetherthere is an interstitial material present between the input region and auser's finger detected on the input region in accordance with someembodiments.

Method 11000 relates to changing characteristics of a tactile output(e.g., embodied in a tactile output profile of the tactile output,including amplitude, pattern, frequency, accompanying audio,coordination between other concurrently generated tactile outputs, etc.)in accordance with whether there is an interstitial material presentbetween the input object (e.g., the user's finger) and the input regionof the device on which the input is detected. Changing thecharacteristics of the tactile output in accordance withpresence/absence of the interstitial material and, optionally, inaccordance with the properties of the interstitial material enables thedevice to provide consistent haptic feedback to the user when the deviceis operated under a variety conditions (e.g., wrapped in various typesof cases, operated by gloved fingers, or operated through clothing).Providing consistent haptic feedback under different operatingconditions enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user providerequired inputs to achieve an intended outcome, and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.}

The method 11000 is performed at an electronic device (e.g., device 300,FIG. 3, or portable multifunction device 100, FIGS. 1A and 4A) (e.g., atablet-shaped device that has a flat front-side (e.g., display-side), aflat back-side (e.g., back-side), and one or more flat, beveled, orcurved peripheral-sides (e.g., four narrow flat faces joined by sharpstraight edges, four narrow flat faces joined by rounded edges, or acontinuous curved surface) that join the front and back sides, withsharp or rounded edges) with a display (e.g., a touch-screen displaythat is disposed on a front-side of the device) (e.g., a touch-screendisplay that occupies substantially the entire front-side of thedevice), an input region that is distinct from the display (e.g., theinput region is the side button xxx that is disposed on one or moreperipheral-sides of the device adjacent to a front-side of the device(e.g., the peripheral-sides include one or more of a leftperipheral-side, a right peripheral-side, a top peripheral-side, or abottom peripheral-side of the device)), and one or more tactile outputgenerators for generating tactile outputs. In some embodiments, theinput region is a touch-sensitive surface that detects intensity of acontact with the touch-sensitive surface, e.g., based on capacitiveand/or resistive touch sensing technologies. In some embodiments, theinput region is a touch-sensitive surface that detects presence of acontact and/or pressing of an input object (e.g., a finger) based onforce, pressure, and/or vibration sensors coupled to the surface. Insome embodiments, the input region is a surface that is coplanar withthe display (e.g., a touch-screen display). In some embodiments, theinput region is a surface (e.g., a protruding region, a recessed region,or a region that is flush with the surrounding surface) that is disposedon a peripheral-side of the device adjacent to the front-side of thedevice (e.g., a left edge, a right edge, a top edge, or a bottom edge,or along two or more of those edges). In some embodiments, theelectronic device further includes one or more first sensors to detectpresence and/or intensities of contacts with the display of the device(e.g., capacitive and/or resistive sensors to detect intensities of acontact on the touch-screen display), and one or more second sensors todetect presence and/or intensities of contacts with first input regionof the device, (e.g., capacitive, resistive, pressure, force, and/orvibration sensors to detect presence, activation, and, optionally,intensities of contact with the first input region (e.g., the side“button” region)). In some embodiments, the device further includes oneor more tactile output generators for generating tactile outputs (e.g.,button actuators for generating localized tactile outputs at the firstinput region, and/or one or more internal actuators for generatingwhole-device tactile outputs, etc.).

In accordance with method 11000 and with reference to FIG. 11A, thedevice detects (11002) a press input on the input region with a finger.In accordance with method 11000 and with reference to FIG. 11A, inresponse to detecting the press input (e.g., detecting an increase inintensity of the contact by the finger above a first intensity threshold(e.g., the contact detection threshold, a light press intensitythreshold that is greater than the contact detection threshold, and adeep press intensity threshold that is greater than the light pressintensity threshold)) on the input region (11004): in accordance with adetermination that the input region is not separated from the finger byan interstitial material (e.g., when the input region is directlyexposed to the skin of the finger, without a case or clothing (e.g.,made of fabric, plastic, leather, silicone, wool, wood, metal, etc.)separating the portion of the device that includes the input region fromthe skin of the finger), the device generates a first tactile output(e.g., tactile output 649 in FIG. 5I1, tactile output 667 in FIG. 5I2,tactile output 679 in FIG. 5I3, tactile output 693 in FIG. 5I4, etc.);and in accordance with a determination that the input region isseparated from the finger by a first interstitial material (e.g., afirst fabric, plastic, leather, silicone, wood, metal, etc. that is usedin a case encasing the portion of the device that includes the inputregion or in a glove encasing the finger), the device generates a secondtactile output (e.g., tactile outputs 653 and 657 in FIG. 5I1, tactileoutputs 671 and 675 in FIG. 5I2, tactile outputs 683 and 687 in FIG.5I3, tactile outputs 697 and 701 in FIG. 5I4, respectively) that isdifferent from the first tactile output (e.g., tactile output 649 inFIG. 5I1, tactile output 667 in FIG. 5I2, tactile output 679 in FIG.5I3, tactile output 693 in FIG. 5I4, respectively) (e.g., the secondtactile output is adapted to create a similar haptic sensation on theuser's finger through the interstitial material of the case or glove).

In accordance with method 11000 and with reference to FIG. 11A, inresponse to detecting the press input on the input region, the deviceperforms (11006) an operation that corresponds to the press input (e.g.,displaying a menu of selectable options on the display, providing adevice status for a first attribute of the device (e.g., ringer volume,availability of unread notifications, etc.)). Performing the sameoperation in response to the press input on the input region, inconjunction with generating a respective tactile output with a tactileoutput profile that is selected in accordance with whether there is aninterstitial material present between the input region and the finger onthe input region, enhances the operability of the device and makes thedevice-user interface more efficient (e.g., by helping the user providerequired inputs to achieve an intended outcome, and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 11000 and with reference to FIG. 11A,generation of the first and second tactile outputs is performed inaccordance with a determination that a characteristic intensity of thepress input exceeds a first intensity threshold (e.g., a light pressintensity threshold); and the method further includes (11008): inresponse to detecting the press input: in accordance with adetermination that the characteristic intensity of the press input doesnot exceed the first intensity threshold, forgoing generation of thefirst tactile output and forgoing generation of the second tactileoutput (e.g., the device does not generate any tactile output inresponse to the press input before the first intensity threshold is met,and does not perform the operation that corresponds to the press input).Forgoing generation of the tactile output when the characteristicintensity of the input does not exceed the first intensity thresholdenhances the operability of the device and makes the device-userinterface more efficient (e.g., by reducing unnecessary distraction tothe user, helping the user provide required inputs to achieve anintended outcome, and reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In accordance with method 11000 and with reference to FIG. 11A, thedevice selects (11010) a second tactile output profile for the secondtactile output profile such that the movement of the first interstitialmaterial (e.g., a portion of the first interstitial material proximateto or adjacent to the input region) generated by the second tactileoutput is substantially similar to movement of the input region on thedevice generated by the first tactile output (e.g., the second tactileoutput produces a similar haptic sensation (e.g., as measured byvibration strengths, frequencies, etc.) on the finger through the firstinterstitial material as that produced by the first tactile outputdirectly on the finger). By selecting the tactile output profile for atactile output generated with the presence of an interstitial materialin a such a way that results in consistent haptic sensations on theuser's finger (e.g., that results in movement of the interstitialmaterial over the input region that is substantially similar to themovement of the input region without the presence of the interstitialmaterial) enhances the operability of the device and makes thedevice-user interface more efficient (e.g., by providing improvedfeedback to help the user provide required inputs to achieve an intendedoutcome, and reducing user mistakes when operating/interacting with thedevice), which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In accordance with method 11000 and with reference to FIG. 11A,selecting the second tactile output profile for the second tactileoutput profile includes (11012) selecting a respective material-specifictactile output profile (e.g., different second tactile output profilesfor the second tactile output depending on the first interstitialmaterial (e.g., leather vs. plastic vs. silicone)) based on the firstinterstitial material and a first tactile output profile thatcorresponds to the first tactile output. In some embodiments, thematerial is determined by the device based on sensors on the device thatsense characteristics of the first interstitial material. In someembodiments, the material is determined by the device based oninformation encoded in the first interstitial material (e.g., a barcode,QR code, or RFID or NFC information embedded in the first interstitialmaterial). In some embodiments, the material is determined based on userinputs at the device (e.g., selection of a type of device case materialfrom a list of options in the user interface). Selecting the tactileoutput profile for a tactile output generated with the presence of aninterstitial material in accordance with the interstitial material andthe tactile output profile for a tactile output generated without theinterstitial material allows the device to provide consistent hapticfeedback to the user under a variety of operating conditions (e.g., withdifferent case materials and glove materials between the input regionand the user's finger), which enhances the operability of the device andmakes the device-user interface more efficient (e.g., by helping theuser provide required inputs to achieve an intended outcome, andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 11000 and with reference to FIG. 11B,selecting the second tactile output profile for the second tactileoutput profile includes (11014) selecting different tactile outputprofiles in accordance with a determination of whether the firstinterstitial material is in a case enclosing at least a portion of thedevice including the input region or the first interstitial material isin a glove enclosing the finger on the input region. This is illustratedin FIGS. 5I1 and 5I2, for example. Selecting the tactile output profilefor a tactile output generated with the presence of an interstitialmaterial in accordance with whether the interstitial material is a caseover the device or a glove over a user's finger allows the device toprovide consistent haptic feedback to the user under a variety ofoperating conditions (e.g., with case or with glove between the inputregion and the user's finger), which enhances the operability of thedevice and makes the device-user interface more efficient (e.g., byhelping the user provide required inputs to achieve an intended outcome,and reducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 11000 and with reference to FIG. 11B, thefirst tactile output and the second tactile output are (11016) generatedby a first tactile output generator of the one or more tactile outputgenerators that resides at the input region and produces localizedtactile outputs at the input region (e.g., as opposed to creatingtactile outputs in many different locations on the device at the sametime, such as via a tactile output generator that is coupled to ahousing of the device). In some embodiments, the first tactile outputgenerator is a surface oscillator that rocks back and forth around afixed pivot underneath the input region or other tactile outputgenerator that is capable of generating tactile outputs that aredirected specifically toward the input region. In some embodiments, thefirst tactile output generator provides localized vibrations at the sidebutton region and not the device as a whole. The localized vibrationshave varying amplitudes at different locations on the device withgreater amplitudes on the side button region where a user's finger willtypically rest while operating the side button region and with loweramplitudes at other places outside of the side button region where auser's hand will not typically rest while operating the side buttonregion. Generating localized tactile output using a tactile outputgenerator that resides at the input region as feedback for a press inputon the input region allows the device to provide consistent hapticfeedback to directly to the user's finger under a variety of operatingconditions (e.g., with case or with glove between the input region andthe user's finger), which enhances the operability of the device andmakes the device-user interface more efficient (e.g., by helping theuser provide required inputs to achieve an intended outcome, andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 11000 and with reference to FIG. 11B, thefirst material includes (11018) a material of a case that encases atleast a portion of the device that includes the input region. This isillustrated in FIG. 5I1, for example. Providing consistent hapticfeedback with or without a case encasing the device enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user provide required inputs to achievean intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 11000 and with reference to FIG. 11B, thefirst material includes (11020) a material of a glove that encases thefinger that is on the input region. This is illustrated in FIG. 5I2, forexample. Providing consistent haptic feedback with or without a gloveover the user's finger on the input region enhances the operability ofthe device and makes the user-device interface more efficient (e.g., byhelping the user provide required inputs to achieve an intended outcome,and reducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 11000 and with reference to FIG. 11B, thedevice determines (11022) whether there is an interstitial materialbetween the input region and the finger in accordance with a settingsparameter specified by a user. Utilizing a settings parameter specifiedby a user is a simple and accurate way of determining whether there isan interstitial material between the input region and the user's finger.Accurate and adaptable determination of the presence of an interstitialmaterial that is widely applicable to indefinite number of user's usagescenarios of their devices enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser provide required inputs to achieve an intended outcome, andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 11000 and with reference to FIG. 11B, thedevice determines (11024) whether there is an interstitial materialbetween the input region and the finger in accordance with a capacitancecharacteristic of the press input provided by one or more sensorslocated at the input region. Utilizing a capacitance characteristic ofthe press input to determine whether there is an interstitial materialpresent between the input region and the user's finger is automatic andeasily adaptable to a wide variety of interstitial material types.Utilizing the capacitance characteristic of the press input to determinewhether there is an interstitial material between the input region andthe finger enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user providerequired inputs to achieve an intended outcome, and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

It should be understood that the particular order in which theoperations in FIGS. 11A-11B have been described is merely exemplary andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 6000, 7000, 8000, 9000, 10000, and 12000) are also applicable inan analogous manner to method 11000 described above with respect toFIGS. 11A-11B. For example, the contacts, gestures, user interfaceobjects, tactile outputs, audio outputs, intensity thresholds, timethresholds, focus selectors, and animations described above withreference to method 11000 optionally have one or more of thecharacteristics of the contacts, gestures, user interface objects,tactile outputs, audio outputs, intensity thresholds, time thresholds,focus selectors, and animations described herein with reference to othermethods described herein (e.g., methods 6000, 7000, 8000, 9000, 10000,and 12000). For brevity, these details are not repeated here.

FIGS. 12A-12E are flow diagrams illustrating a method 12000 of providingdifferent tactile outputs for distinct types of inputs detected on anoff-display input region disposed on a peripheral side of the device inaccordance with some embodiments.

Method 12000 relates to generating tactile outputs with differenttactile output profiles in responds to different types of touch inputsdetected on an off-screen input region on a peripheral-side of thedevice adjacent to the display, including a press input on the inputregion and a swipe input across the input region (e.g., a swipe input ina horizontal direction across an elongated input region or a swipe inputin a vertical direction along the elongated input region). Providingdifferent haptic feedback (via tactile outputs with different tactileoutput profiles) in response to different types of inputs detected onthe input region enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user providerequired inputs to achieve an intended outcome, and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.}

The method 12000 is performed at an electronic device (e.g., device 300,FIG. 3, or portable multifunction device 100, FIGS. 1A and 4A) (e.g., atablet-shaped device that has a flat front-side (e.g., display-side), aflat back-side (e.g., back-side), and one or more flat, beveled, orcurved peripheral-sides (e.g., four narrow flat faces joined by sharpstraight edges, four narrow flat faces joined by rounded edges, or acontinuous curved surface) that join the front and back sides, withsharp or rounded edges) with a display that is disposed on a front-sideof the device (e.g., a touch-screen display that occupies substantiallythe entire front-side of the device, a touch-screen display thatincludes a touch-sensitive surface that is coplanar with the display),an input region (e.g., the side button 455, 456, and/or 457) that isdisposed on one or more peripheral-sides of the device adjacent to thefront-side of the device (e.g., the peripheral-sides include one or moreof a left peripheral-side, a right peripheral-side, a topperipheral-side, or a bottom peripheral-side of the device), and one ormore tactile output generators for generating tactile outputs (e.g.,button actuators for generating localized tactile outputs at the firstinput region, and/or one or more internal actuators for generatingwhole-device tactile outputs, etc.). In some embodiments, the inputregion is a touch-sensitive surface that detects intensity of a contactwith the touch-sensitive surface, e.g., based on capacitive and/orresistive touch sensing technologies. In some embodiments, the inputregion is a touch-sensitive surface that detects presence of a contactand/or pressing of an input object (e.g., a finger) based on force,pressure, and/or vibration sensors coupled to the surface. In someembodiments, the input region is a surface that is coplanar with thedisplay (e.g., a touch-screen display). In some embodiments, the inputregion is a surface (e.g., a protruding region, a recessed region, or aregion that is flush with the surrounding surface) that is disposed on aperipheral-side of the device adjacent to the front-side of the device(e.g., a left edge, a right edge, a top edge, or a bottom edge, or alongtwo or more of those edges). In some embodiments, the electronic devicefurther includes one or more first sensors to detect presence and/orintensities of contacts with the display of the device (e.g., capacitiveand/or resistive sensors to detect intensities of a contact on thetouch-screen display), and one or more second sensors to detect presenceand/or intensities of contacts with first input region of the device,(e.g., capacitive, resistive, pressure, force, and/or vibration sensorsto detect presence, activation, and, optionally, intensities of contactwith the first input region (e.g., the side “button” region)).

In accordance with method 12000 and with reference to FIG. 12A, thedevice detects (12002) an input on the input region (e.g., detecting aninput by a contact on the input region). For example, the input is oneof a press input (e.g., input by contact 506, input by contact 518,input by contact 530, input by contact 534, input by contacts 542 and544, input by contact 552, input by contact 570, input by contact 594,input by contact 579, input by contact 565, input by contact 557, inputby contact 509, input by contact 605, input by contact 627), ahorizontal swipe input (e.g., input by contact 585), or a vertical swipeinput (e.g., input by contact 564, input by contact 580, input bycontact 551, input by contact 547), as illustrated in FIGS. 5A1-5A8,5B1-5B11, 5C1-5C20, 5D1-5D24, 5E1-5E5, 5F1-5F6, 5G1-5G5, and 5H1-5H5.

In response to detecting the input on the input region (12004): inaccordance with a determination that the input by the contact (e.g.,light press input by contact 506 in FIG. 5A5, light press input bycontact 518 in FIG. 5B3, light press input by contact 534 in FIG. 5B7,light press input by contact 552 in FIG. 5C6, light press input bycontact 557 in FIG. 5D17, light press input by contact 509 in FIG. 5F3,or light press input by contact 627 in FIG. 5H2) on the input regionmeets first criteria (e.g., button-press criteria), wherein the firstcriteria (e.g., the button-press criteria) require an increase inintensity of the input above a first intensity threshold (e.g., a lightpress intensity threshold that is greater than a contact detectionthreshold and less than a deep press intensity threshold) in order forthe first criteria to be met, the device performs a first operation(e.g., displaying a menu of selectable options, displaying a controlobject for controlling a device parameter (e.g., volume, ringer setting,etc.)) and generating a first tactile output (e.g., tactile output 510in FIG. 5A5, tactile output 524 in FIG. 5B3, tactile output 536 in FIG.5B7, tactile output 566 in FIG. 5C6, tactile output 553 in FIG. 5D17,tactile output 625 in FIG. 5F3, or tactile output 629 in FIG. 5H2) witha first tactile output profile (e.g., to simulate actuation of a buttonin a direction perpendicular to a region of input region at the contactlocation). In some embodiments, the first tactile output is generatedduring a down-press phase of the input. In some embodiments, the firsttactile output is generated during a release phase of the input. In someembodiments, a pair of tactile outputs are generated, one for thedown-press and one for the release.

In response to detecting the input on the input region (12004): inaccordance with a determination that the input on the input region(e.g., input by contact 585 in FIGS. 5D2-5D3, input by contact 564 inFIGS. 5C10-5C12, input by contact 580 in FIG. 5C15-5C17, input bycontact 551 in FIGS. 5D18-5D19, input by contact 547) meets secondcriteria that are distinct from the first criteria (e.g., the secondcriteria are one of button-toggle criteria or button-slide criteria),wherein the second criteria (e.g., one of button-toggle criteria orbutton-slide criteria) require that the input includes movement of acontact along a first direction across the input region (e.g., in adirection across an elongated touch-sensitive region on theperipheral-side of the device (for button-toggle criteria), or in adirection along the elongated touch-sensitive region on theperipheral-side of the device (for button-slide criteria)) in order forthe second criteria to be met, the device performs a second operation(e.g., turning on/off the ringer) different from the first operation andgenerating a second tactile output (e.g., tactile output 581 in FIG.5D3, tactile output 5663 in FIGS. 5C10-5C12, tactile outputs 586 and 592in FIGS. 5C15-5C17, tactile outputs 549 and 545 in FIGS. 5D18-5D19) witha second tactile output profile that is distinct from the first tactileoutput profile (e.g., a tactile output to simulate toggling of a switchthat toggles in response to lateral movement or force (e.g., in abutton-toggle input), or a tactile output to simulate sliding of aslider that moves in response to lateral movement or force (e.g., in abutton slide input)).

In accordance with method 12000 and with reference to FIG. 12A, inresponse to detecting the input on the input region: in accordance witha determination that the input on the input region (e.g., input bycontact 585 in FIGS. 5D2-5D3, input by contact 564 in FIGS. 5C10-5C12,input by contact 580 in FIG. 5C15-5C17, input by contact 551 in FIGS.5D18-5D19, input by contact 547) meets third criteria that are distinctfrom the first criteria and the second criteria (e.g., the thirdcriteria are the other one of the button-toggle criteria or button-slidecriteria), wherein the third criteria require that the input includesmovement of a contact along a second direction distinct from (e.g.,orthogonal to) the first direction across the input region in order forthe third criteria to be met, the device performs (12006) a thirdoperation (e.g., turning on/off the display, activating voice-baseddigital assistant, start a shut-down process) that is distinct from thefirst operation and the second operation, and the device generates athird tactile output (e.g., tactile output 581 in FIG. 5D3, tactileoutput 5663 in FIGS. 5C10-5C12, tactile outputs 586 and 592 in FIGS.5C15-5C17, tactile outputs 549 and 545 in FIGS. 5D18-5D19) with a thirdtactile output profile that is distinct from the first tactile outputprofile and the second tactile output profile (e.g., a tactile output tosimulate the other one of the button toggle or button slide). Providingdifferent haptic feedback (via tactile outputs with different tactileoutput profiles) in response to different types of inputs detected onthe input region (including a press input, a swipe input along the inputregion, and a swipe input across the input region) enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user provide required inputs to achievean intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 12000 and with reference to FIG. 12A, inresponse to detecting the input on the input region: in accordance witha determination that the input on the input region (e.g., deep pressinput by contact 506 in FIG. 5A6, or deep press input by contact 534 inFIG. 5B9) meets fourth criteria (e.g., deep press criteria) that aredistinct from the first criteria and the second criteria (and the thirdcriteria), wherein the fourth criteria (e.g., deep press criteria)require an increase in intensity of the input above a second intensitythreshold (e.g., a deep press intensity threshold) that is greater thanthe first intensity threshold (e.g., the light press intensitythreshold) in order for the fourth criteria to be met, the deviceperforms (12008) a fourth operation (e.g., start a warning for animpending operation) that is distinct from the first operation and thesecond operation (and the third operation), and the device generates afourth tactile output (e.g., tactile output 512 in FIG. 5A6, or tactileoutput 538 in FIG. 5B9) with a fourth tactile output profile that isdistinct from the first tactile output profile and the second tactileoutput profile (and the third tactile output profile) (e.g., the fourthtactile output is a tactile output to simulate a stronger button clickthan the regular button click triggered by a light press input).Providing different haptic feedback (via tactile outputs with differenttactile output profiles) in response to different types of inputsdetected on the input region (including a press input, a deep pressinput, and a swipe input) enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser provide required inputs to achieve an intended outcome, andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 12000 and with reference to FIG. 12B, inresponse to detecting the input on the input region: in accordance witha determination that the input on the input region (e.g., input bycontact 506 in FIG. 5A3) meets fifth criteria (e.g., button-touchcriteria) that are distinct from the first criteria and the secondcriteria (and the third criteria and the fourth criteria), wherein thefifth criteria (e.g., button-touch criteria) require an intensity of theinput remain below the first intensity threshold (e.g., a light pressintensity threshold) (e.g., the fifth criteria are met when the usersimply rest his/her finger on the input region so that the contactintensity is above the contact detection intensity threshold) in orderfor the fifth criteria to be met, the device performs (12010) a fifthoperation (e.g., show a hint of a menu) that is distinct from the firstoperation and the second operation (and the third operation and thefourth operation), and (optionally) the device generates a fifth tactileoutput (e.g., tactile output 508 in FIG. 5A3) with a fifth tactileoutput profile that is distinct from the first tactile output profileand the second tactile output profile (and the third tactile outputprofile and the fourth tactile output profile) (e.g., the fifth tactileoutput is a tactile output to simulate a texture for the user to knowwhere the input region is located). In some embodiments, the deviceforgoes generation of the fifth tactile output when the fifth criteriaare met. Providing different haptic feedback (via tactile outputs withdifferent tactile output profiles) in response to different types ofinputs detected on the input region (including a press input, a swipeinput, and a light touch input or finger rest input) enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user provide required inputs to achievean intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 12000 and with reference to FIG. 12B, inresponse to detecting the input on the input region: in accordance witha determination that the input on the input region (e.g., input bycontact 518 in FIG. 5B3) meets sixth criteria (e.g., upper-button-presscriteria) that are distinct from the first criteria and the secondcriteria (and the third criteria, the fourth criteria, and the fifthcriteria), wherein the sixth criteria require that a contact ismaintained at a first predefined region of the input region (e.g., anupper half of the side button) and that an intensity of the contactincreases above the first intensity threshold (e.g., the light pressintensity threshold) in order for the sixth criteria to be met, thedevice performs (12012) a sixth operation (e.g., increasing a value of acontrol parameter (e.g., volume, brightness, etc.)) that is distinctfrom the first operation and the second operation (and the thirdoperation, the fourth operation, and the fifth operation), and thedevice generates a sixth tactile output (e.g., tactile output 524 inFIG. 5B3) with a sixth tactile output profile that is distinct from thefirst tactile output profile and the second tactile output profile (andthe third tactile output profile, the fourth tactile output profile, andthe fifth tactile output profile) (e.g., the sixth tactile output is atactile output to simulate pressing on an upper portion of a rockerswitch that ratchets up). In accordance with a determination that theinput on the input region (e.g., input by contact 534 in FIG. 5B7) meetsseventh criteria (e.g., lower-button-press criteria) that are distinctfrom the first criteria, the second criteria, (the third criteria, thefourth criteria, the fifth criteria) and the sixth criteria, wherein theseventh criteria require that a contact is maintained at a secondpredefined region of the input region that is below the first predefinedregion (e.g., the second predefined region is a lower half of the sidebutton) and that an intensity of the contact increases above the firstpredefined intensity threshold (e.g., the light press intensitythreshold) in order for the seventh criteria to be met, the deviceperforms a seventh operation (e.g., decreasing a value of the controlparameter (e.g., volume, brightness, etc.)) that is distinct from thefirst operation and the second operation, (the third operation, thefourth operation, and the fifth operation) and the sixth operation, andthe device generates a seventh tactile output (e.g., tactile output 536in FIG. 5B7) with a seventh tactile output profile that is distinct fromthe first tactile output profile and the second tactile output profile(and the third, fourth, fifth tactile output profile, and optionally,the sixth tactile output profile) (e.g., the seventh tactile output is atactile output to simulate pressing on a lower portion of a rockerswitch that ratchets down). For example, pressing on the top portion ofthe side button causes the volume to increase, and generation of aseries of tactile output that increases in magnitude in accordance withthe increasing volume values; while pressing on the bottom portion ofthe side button causes the volume to decrease, and generation of aseries of tactile output that decreases in magnitude in accordance withthe decreasing volume values. Providing different haptic feedback (viatactile outputs with different tactile output profiles) in response totouch inputs detected on different sub-regions of the input regionenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user provide requiredinputs to achieve an intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 12000 and with reference to FIG. 12C, inresponse to detecting the input on the input region: in accordance witha determination that the input on the input region (e.g., input bycontacts 542 and 544 in FIGS. 5B10-5B11) meets eighth criteria (e.g.,simultaneous-button-press criteria) that are distinct from the firstcriteria and the second criteria (and the third criteria, the fourthcriteria, the fifth criteria), and the sixth criteria and the seventhcriteria, wherein the eighth criteria require that a first contact ismaintained at the first predefined region of the input region (e.g., anupper half of the side button) with an intensity of the first contactabove the first intensity threshold (e.g., the light press intensitythreshold), while a second contact is maintained at the secondpredefined region of the input region with an intensity of the secondcontact above the first predefined intensity threshold (e.g., the lightpress intensity threshold) in order for the eighth criteria to be met,the device performs (12014) an eighth operation (e.g., switching tocontrolling a different parameter using the side button) that isdistinct from the first operation, the second operation, (the thirdoperation, the fourth operation, the fifth operation,) and the sixthoperation and the seventh operation, and the device generates an eighttactile output (e.g., tactile outputs 546 and 548 in FIG. 5B11) with aneighth tactile output profile that is distinct from the first tactileoutput profile and the second tactile output profile (and the third,fourth, fifth tactile output profile, and the sixth and seventh tactileoutput profiles) (e.g., the eighth tactile output is a tactile output tosimulate pressing on two buttons together). Providing different hapticfeedback (via tactile outputs with different tactile output profiles) inresponse to respective press inputs detected on different sub-regions ofthe input region and a press input concurrently detected on bothsub-regions of the input region enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by helping theuser provide required inputs to achieve an intended outcome, andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 12000 and with reference to FIG. 12C, inresponse to detecting the input on the input region: in accordance witha determination that the input (e.g., input by contact 506 in FIG. 5A3or input by contact 627 in FIG. 5H3) on the input region meets ninthcriteria (e.g., long-press criteria) that are distinct from the firstcriteria and the second criteria (and the third, fourth, fifth, sixth,seventh, and eight criteria), wherein the ninth criteria (e.g.,long-press criteria) require that a contact is maintained on the inputregion for at least a threshold amount of time (e.g., 3 seconds) withless than a threshold amount of movement (e.g., the ninth criteria aremet by a touch-hold gesture on the side button) in order for the ninthcriteria to be met, the device performs (12016) a ninth operation (e.g.,activating a voice-based digital assistant) that is distinct from thefirst operation and the second operation (and the third, fourth, fifth,sixth, seventh, and eighth operations), and the device generates a ninthtactile output (e.g., tactile output 508 in FIG. 5A3, or tactile output631 in FIG. 5H3) with a ninth tactile output profile that is distinctfrom the first tactile output profile and the second tactile outputprofile (and the third, fourth, fifth, sixth, seventh, and eighthtactile output profiles). Providing different haptic feedback (viatactile outputs with different tactile output profiles) in response todifferent types of inputs detected on the input region (including apress input, a swipe input, and a long press input) enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user provide required inputs to achievean intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 12000 and with reference to FIG. 12C, thedevice detects (12018) termination of the input on the input region. Inresponse to detecting the termination of the input: in accordance with adetermination that a tactile output has been generated in response tothe input based on satisfaction of one or more intensity-based criteria(e.g., button-activation criteria, deep-press criteria, long-presscriteria, simultaneous-button-press criteria, etc.) by the input, thedevice generates a respective button-release tactile output with arespective button-release tactile output profile. In some embodiments,different button-release tactile output profiles are used for differenttactile outputs that were generated in response to the input. Forexample, the button-release tactile output profile for a deep pressinput is different from the button-release tactile output profile for alight press input on the side button. Generating a respectivebutton-release tactile output in accordance with a determination that atactile output has been generated based on satisfaction of one or moreintensity-based criteria (e.g., for a finger rest input, a light pressinput, a deep press input, a two-finger press input, etc.) enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by notifying the user that a termination of the inputhas been detected, helping the user provide required inputs to achievean intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 12000 and with reference to FIG. 12D,detecting the input on the input region includes detecting an increasein the intensity of the input prior to detecting the intensity of theinput increasing above the first intensity threshold, and wherein themethod includes (12020): in response to detecting the increase in theintensity of the input prior to detecting the intensity of the inputincreasing above the first intensity threshold: displaying a visualfeedback on the display with at least one visual characteristic (e.g.,visibility of wake screen user interface 501 in FIG. 5A4, amount of menu556 that is revealed in FIGS. 5C3-5C5) that varies by an amount inaccordance with a current intensity of the input on the input region(e.g., revealing a user interface object on the display by an amountthat varies in accordance with a current intensity of the first input).In some embodiments, as the intensity of the contact increases on thefirst input region (e.g., the side button), the first user interfaceobject (e.g., a menu, a clipboard) slides gradually onto the displayfrom the edge of the display that is adjacent to the first input region(e.g., the side button). In some embodiments, as the intensity of thefirst contact increases on the first input region (e.g., the sidebutton), the first user interface object (e.g., the menu or theclipboard) fades in gradually onto the display. In some embodiments, theamount by which the user interface object is visible on the display isvariable as the intensity of the contact varies (e.g., increases and/ordecreases) before reaching a second intensity threshold that is greaterthan the first intensity threshold. Once the intensity of the contactincreases above a second intensity threshold, the user interface objectis fully displayed. In some embodiments, the direction in which the userinterface slides onto the display is the same as or parallel to thedirection of the force/pressure exerted by the finger on the first inputregion. Providing dynamic visual feedback that changes in accordance anintensity of the input on the input region before the input meets theintensity threshold of the first criteria enhances the operability ofthe device and makes the user-device interface more efficient (e.g., byproviding improved feedback to help the user provide required inputs toachieve an intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 12000 and with reference to FIG. 12D,displaying the visual feedback on the display includes (12022) revealinga user interface object (e.g., menu 556 that is revealed in FIGS.5C3-5C5, or menu 577 in FIGS. 5D4-5D5) with movement of the userinterface object onto the display in a direction that corresponds to adirection of force exerted on the input region (e.g., the direction inwhich the user interface slides onto the display is the same as orparallel to the direction of the force/pressure exerted by the finger onthe first input region. Revealing a user interface object with movementof the user interface object onto the display in a direction thatcorresponds to a direction of force exerted on the input region enhancesthe operability of the device and makes the user-device interface moreefficient (e.g., by providing improved feedback to help the user providerequired inputs to achieve an intended outcome, and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 12000 and with reference to FIG. 12D, inaccordance with a determination that the input meets the second criteria(e.g., button toggle criteria or button-slide criteria), performing thesecond operation and generating the second tactile output with thesecond tactile output profile, includes (12024): determining whether thesecond operation is for changing a value for a device attribute from afirst value to a second value that is distinct from the first value(e.g., from ON to OFF, from a low value to a high value, etc.), or fromthe second value to the first value (e.g., from OFF to ON, from a highvalue to a low value, etc.); and in accordance with a determination thatthe second operation is for changing the value for the device attributefrom the first value to the second value (e.g., from ON to OFF, orincreasing the value of the device attribute), selecting a first set ofparameter values (e.g., first frequency, first amplitude, first pattern,etc.) for the second tactile output profile; and in accordance with adetermination that the second operation is for changing the value forthe device attribute from the second value to the first value (e.g.,from OFF to ON, or decreasing the value of the device attribute),selecting a second set of parameter values (e.g., second frequency,second amplitude, second pattern, etc.) for the second tactile outputprofile. For example, a similar tactile output is used to simulate thehaptic sensation produced by a toggling switch that is flipped; butdepending on the actual state of the controlled device attribute that ischanged, some (not all) aspects of the second tactile output are changed(e.g., amplitude, but not pattern; or pattern but not amplitude) inaccordance with the direction that the state is changed by the input. Inanother example, a similar tactile output is used to simulate the hapticsensation produced by mechanical slider that is moved; but depending onthe actual direction that the value of the controlled device attributeis changed, some (not all) aspects of the second tactile output arechanged (e.g., amplitude, but not pattern; or pattern but not amplitude)in accordance with the direction that the value is changed by the input.Providing different haptic feedback (via tactile outputs with differenttactile output profiles) depending on the direction of change in adevice attribute value (e.g., from value 1 to value to vs. from value 2to value 1) resulted from a swipe input (e.g., a swipe in one directionvs. a swipe in an opposite direction) enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user provide required inputs to achieve an intended outcome,and reducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 12000 and with reference to FIG. 12D, thedevice includes (12026) intensity sensors (e.g., capacitive, resistive,force, pressure, and/or vibration sensors) for detecting intensities ofcontacts on the input region, wherein detection of the intensities ismade even before actuation of the input region is detected (e.g.,actuation of the input region is not dependent upon a completion of acircuit by collapsing a physical mechanism that provides resistance suchas a dome switch or other mechanical switch mechanism). For example, theincrease in intensity of the contact does not actuate the surface of theside button in the direction of the press input, and detecting theintensity relative to an intensity threshold does not depend on acomparison between a physical movement of the surface of the side buttonto a movement threshold. The generation of the first tactile output inresponse to the press input is not in accordance with a movementthreshold being met by the movement of the surface of the as a result ofthe press input. Using intensity sensors to detect intensities ofcontacts on the input region even before physical actuation of the inputregion is detected allows the device to detect inputs on the inputregion more quickly and more accurately, and makes the device moreresponsive to user inputs. Utilizing input regions with these intensitysensors (instead of physically actuated hardware buttons) to detectinputs enhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user provide requiredinputs to achieve an intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 12000 and with reference to FIG. 12E, thedevice includes (12028) movement sensors (e.g., capacitive, resistive,force, pressure, and/or vibration sensors) for detecting movement ofcontacts on the input region, wherein detection of the movement of thecontacts is made before actuation of the input region is detected (e.g.,actuation of the input region is not dependent upon a completion of acircuit by collapsing a physical mechanism that provides resistance suchas a dome switch or other mechanical switch mechanism). For example, themovement of the contact along the first direction is relative to theinput region, and does not cause corresponding displacement of the inputregion in the first direction in accordance with the movement of thecontact in the first direction. The generation of the second tactileoutput is not in accordance with a movement threshold being met by thedisplacement of any part of the input region as a result of the movementof the contact. Using movement sensors to detect movement of contacts onthe input region even before physical actuation of the input region isdetected allows the device to detect inputs on the input region morequickly and more accurately, and makes the device more responsive touser inputs. Utilizing input regions with these movement sensors(instead of physically actuated joysticks or trackballs) to detectinputs enhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user provide requiredinputs to achieve an intended outcome, and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In accordance with method 12000 and with reference to FIG. 12E, thedevice includes (12030) movement sensors (e.g., capacitive, resistive,force, pressure, and/or vibration sensors) for detecting movement ofcontacts on the input region, wherein the first and second tactileoutputs (and other localized button tactile outputs) are generated bymovement of at least a portion of the input region with a movementpattern that is distinct from the movement of the contacts. For example,the amplitude of the movement of the button is much smaller than themovement of the contact. The button oscillates, and the movement by thecontact does not oscillate or oscillate at a much lower frequency. Usingmovement sensors to detect movement of contacts on the input regionwhich has a different movement pattern from the movement pattern of thephysical actuation of the input region allows the device to detectinputs on the input region more quickly and more accurately, and makesthe device more responsive to user inputs. Utilizing input regions withthese movement sensors (instead of physically actuated joysticks ortrackballs) to detect inputs enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser provide required inputs to achieve an intended outcome, andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In accordance with method 12000 and with reference to FIG. 12E, thefirst criteria and the second criteria (and the third criteria) are met(12032) without requiring a focus selector (e.g., a cursor or finger)being concurrently present on the display. Operating the device usinginputs detected on the input region without requiring a focus selectorbeing concurrently present on the display reduces clutter on the userinterface, and free up screen space for additional functionalities andcontrols. Thus, utilizing an off-display input region that is operatedwithout requiring concurrently presence of a focus selector (e.g., acontact or a cursor) on the display enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byproviding additional control options without cluttering the userinterface with additional displayed controls or finger contacts), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In accordance with method 12000 and with reference to FIG. 12E, thefirst and second tactile outputs (and the other localized button tactileoutputs) are (12034) localized on the peripheral-side of the device(e.g., generated by the button actuators located on or proximate to theperipheral-side of the device, as opposed to being propagated from aninternal actuator inside of the device to locations outside of the inputregion (e.g., throughout the device.)). Providing localized tactileoutputs on the peripheral side of the device in response to inputsdetected on the input region on the peripheral-size of the deviceenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by providing improved feedback to helpthe user provide required inputs to achieve an intended outcome, and byreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

It should be understood that the particular order in which theoperations in FIGS. 12A-12E have been described is merely exemplary andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 6000, 7000, 8000, 9000, 10000, and 11000) are also applicable inan analogous manner to method 6000 described above with respect to FIGS.12A-12E. For example, the contacts, gestures, user interface objects,tactile outputs, audio outputs, intensity thresholds, time thresholds,focus selectors, and animations described above with reference to method12000 optionally have one or more of the characteristics of thecontacts, gestures, user interface objects, tactile outputs, audiooutputs, intensity thresholds, time thresholds, focus selectors, andanimations described herein with reference to other methods describedherein (e.g., methods 6000, 7000, 8000, 9000, 10000, and 11000). Forbrevity, these details are not repeated here.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best use the invention and variousdescribed embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method, comprising: at an electronic devicewith a display, and a first input region that is separate from thedisplay: detecting a first portion of a first input on the first inputregion; in response to detecting the first portion of the first input onthe first input region: in accordance with a determination that thefirst portion of the first input meets first criteria, wherein the firstcriteria include a set of criteria that are met when the first input isdetected on the first input region without a characteristic intensity ofthe first input increasing above a first intensity threshold,displaying, on the display, a precursor object of a user interfaceobject that includes one or more selectable options; while displayingthe precursor object on the display, detecting a second portion of thefirst input on first input region, including detecting an increase inintensity of the first input above the first intensity threshold; and inresponse to detecting the increase in intensity of the first inputduring the second portion of the first input, revealing the userinterface object on the display by an amount that varies in accordancewith a current intensity of the first input.
 2. The method of claim 1,including: after detecting the second portion of the first input,detecting a third portion of the first input on the first input region,including detecting a continued increase in intensity of the firstinput; in response to detecting the continued increase in intensity ofthe first input: in accordance with a determination that the thirdportion of the first input meets second criteria, wherein the secondcriteria include a set of criteria that are met when a characteristicintensity of the first input increases above a second intensitythreshold that is greater than the first intensity threshold before anend of the first input is detected: displaying the user interface objectthat includes the one or more selectable options on the display.
 3. Themethod of claim 2, including: after detecting the third portion of thefirst input, detecting a fourth portion of the first input on the firstinput region, including detecting a decrease in intensity of the firstinput: in accordance with a determination that the second criteria weremet before detecting the fourth portion of the first input, maintainingdisplay of the user interface object without altering an amount by whichthe user interface object is currently displayed; and in accordance witha determination that the second criteria were not met before detectingthe fourth portion of the first input, decreasing an amount by which theuser interface object is revealed on the display in accordance with acurrent intensity of the first input.
 4. The method of claim 2,including: in response to detecting the continued increase in intensityof the first input: in accordance with a determination that the thirdportion of the first input meets the second criteria: generating a firsttactile output.
 5. The method of claim 4, wherein the device includes afirst tactile output generator that resides locally at the first inputregion and that provides localized tactile outputs, including the firsttactile output, at the first input region.
 6. The method of claim 4,wherein the generation of the first tactile output is synchronized withsatisfaction of the second criteria by the first input.
 7. The method ofclaim 4, wherein the generation of the first tactile output issynchronized with initial full-display of the user interface object onthe display.
 8. The method of claim 2, wherein the device includes atouch-sensitive surface that is distinct from the first input region,and wherein the method includes: while displaying the user interfaceobject including the one or more selectable options on the display,detecting a second input on the touch-sensitive surface at a locationthat corresponds to a first selectable option of the one or moreselectable options in the user interface object; and in response todetecting the second input: in accordance with a determination that thesecond input meets selection criteria, selecting the first selectableoption.
 9. The method of claim 8, including: in response to detectingthe second input: in accordance with a determination that the secondinput meets selection-confirmation criteria, performing a firstoperation that corresponds to the first selectable option.
 10. Themethod of claim 8, including: in response to detecting the second input:in accordance with a determination that the second input meets dragcriteria, displaying a representation of the first selectable optionoutside of the user interface object.
 11. The method of claim 10,including: in response to detecting the second input: in accordance witha determination that the second input meets drop criteria, displaying acopy of the first selectable option outside of the user interfaceobject.
 12. The method of claim 8, wherein the second input is detectedwhile the user interface object is partially revealed on the display,without the second criteria being met by the first input.
 13. The methodof claim 2, including: while displaying the user interface objectincluding the one or more selectable options on the display, detecting athird input on the first input region, wherein the third input includesmovement of a contact across the first input region; and in response todetecting the third input, scrolling a selection indicator through theone or more selectable options on the display in accordance with themovement of the contact in the third input.
 14. The method of claim 13,including: while the selection indicator is moved to a first selectableoption of the one or more selectable options on the display inaccordance with the movement of the contact in the third input on thefirst input region, detecting a fourth input on the first input regionthat meets selection-confirmation criteria; and in response detectingthe fourth input that meets the selection-confirmation criteria,performing an operation corresponding to the first selectable options.15. The method of claim 2, wherein the device includes a touch-sensitivesurface that is distinct from the first input region, and wherein themethod includes: while displaying the user interface object includingthe one or more selectable options on the display, detecting a fifthinput on the touch-sensitive surface, including: detecting touch-down ofa contact on the touch-sensitive surface at a location that correspondsto a second user interface object that is displayed outside of the userinterface object, detecting movement of the contract across thetouch-sensitive surface to a location within the user interface object,and detecting lift-off of the contact from the touch-sensitive surfacewhile the second user interface is within the user interface object; andin response to detecting the fifth input, adding the second userinterface object as a new selectable option in the user interfaceobject.
 16. The method of claim 2, wherein the device includes atouch-sensitive surface that is distinct from the first input region,and wherein the method includes: while displaying the user interfaceobject including the one or more selectable options on the display,detecting a sixth input on the touch-sensitive surface, including:detecting touch-down of a contact on the touch-sensitive surface at alocation that corresponds to a respective selectable option that isdisplayed within the user interface object, detecting movement of thecontract across the touch-sensitive surface to a location outside of theuser interface object, and detecting lift-off of the contact from thetouch-sensitive surface while the respective selectable option isdragged outside of the user interface object; and in response todetecting the sixth input, removing the respective selectable optionfrom the user interface object.
 17. An electronic device, comprising: adisplay; a first input region that is separate from the display; one ormore processors; memory; and one or more programs, wherein the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: detecting a first portion of a first input on thefirst input region; in response to detecting the first portion of thefirst input on the first input region: in accordance with adetermination that the first portion of the first input meets firstcriteria, wherein the first criteria include a set of criteria that aremet when the first input is detected on the first input region without acharacteristic intensity of the first input increasing above a firstintensity threshold, displaying, on the display, a precursor object of auser interface object that includes one or more selectable options;while displaying the precursor object on the display, detecting a secondportion of the first input on first input region, including detecting anincrease in intensity of the first input above the first intensitythreshold; and in response to detecting the increase in intensity of thefirst input during the second portion of the first input, revealing theuser interface object on the display by an amount that varies inaccordance with a current intensity of the first input.
 18. A computerreadable storage medium storing one or more programs, the one or moreprograms comprising instructions, which when executed by an electronicdevice with a display, a first input region that is separate from thedisplay, one or more processors, and memory, cause the device to: detecta first portion of a first input on the first input region; in responseto detecting the first portion of the first input on the first inputregion: in accordance with a determination that the first portion of thefirst input meets first criteria, wherein the first criteria include aset of criteria that are met when the first input is detected on thefirst input region without a characteristic intensity of the first inputincreasing above a first intensity threshold, display, on the display, aprecursor object of a user interface object that includes one or moreselectable options; while displaying the precursor object on thedisplay, detect a second portion of the first input on first inputregion, including detecting an increase in intensity of the first inputabove the first intensity threshold; and in response to detecting theincrease in intensity of the first input during the second portion ofthe first input, reveal the user interface object on the display by anamount that varies in accordance with a current intensity of the firstinput.